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A tne daf£) nat^erina (or 
intelHo^=‘nc^^) orocesSf tn<=» oecision Tai^ino process/ and the 
oDeratina forces reouired to achieve a oarticular mission. 
Llnfortunatelv/ the majority of t^^e resources soent in the 
imorovenr'ent of these systems are exoended in either 
orovinino better oata aatherinq/ failsafe communications/ or 
better weaoon svstems. Little thought is given to how the 
decision Ta<er orocesses the data in oraer to make effective 
decisions. Several comcuterized decision aids have been 
suaoested to fill this can. One of these aids/ Ooerations 
and Intelligence (OPIdT) is evaluated to assess its utility 
as a CO Ti cuter i/ed decision aid in C3 aoolications. OPIdT is 
an on-linp^ interactive/ real-time decision aid which 
assists decision makers by prescribing a straightforward 
normative croceoure for organizing ana analyzing ciifficult 
decision oroolems. The results of the experiment shew tnat 
OPIdT aids in tne decision making orocess/ but has some 
severe limitations as it currently exists. 
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T. CESCPlPfiriM OF OKCI3ION AN'iLrSlS 



Ourina tiTes of crisis or war/ there is orobablv no 
field of pndeavor like military and strategic command and 
control where decisions are made ouicklv/ but under 
conoitions of risk and extreme uncertainty. This is 
esoecially tru^ since accurate infor'^ation on all ascects of 
a orcolem are almost always concealed from cecision makers. 
Every military force tries to conceal itself and rra<es every 
attemot to deceive the oooosina decision maker as to its 
intentions. Facts surrounding the tactical situation are 
elusive/ esoecially those concerning the enemy. These tyoes 
of decisions/ then/ are difficult and freouently would seem 
to defy a systematic decision making process (Department of 
the Army/ 1^80/ o. 3-2). At the same time/ there is no 
arena in which bad decisions have more tragic results. The 
cost of making errors can grow exponentially mainly due to 
the coTDlex relationsnios and resulting chain reactions/ or 
the errors could cancel themselves out ancj the costs not oe 
immediately known (Turban and Meredith/ 1977/ o. ^). 

Con seauen t 1 y / wnile decision makers must recognize that 
orderly/ rational decision making orocedures are difficult/ 
they must at the same time constantly strive to base 
decisions on the most rational thought orocess oossible. To 
reduce error Drooability/ and for the sake of survival/ 
decision makers must oecome more sophisticated (Turban and 
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^’eredith^ 1*^77^ o. 5). They must learn to utilize ne^ 
tools and techniques that are heina aevelooed. No one could 
imaqine a successful suraeon utilizing equioment ano 
orocedures from the turn of tnp century* Yetf in decision 
making# you can still find aecision makers using the tools 
and techniques of that time. 

Military and strategic decisions are not only made 
under conditions of uncertainty (where not all facts are 
availaole) and stress# they are also rarely considerea 
final. The ev^r changino situation brings with it the 
requirement to continuously rev i se ,aoD ra i sa I s # estimates# 
and oerhaos decisions. This occurs since decision makino is 
based on the future but is deoendent on the oast. For 
years# managers have considered decision makinq to be a oure 
art or talent which is acquired over a long oeriod of time 
through exoerience or trial and error. It has been 
considered an art because a wide variety of individual 
styles can oe used in aooroachino and successfully solving 
tne same tyoe of oroblems. One would n ormally base these 
styles on creativity# judgment# intuition# anq experience 
rather than some sort of systematic method (Turban and 
Meredith, 1977, n. 5). 

• Decision makino as a discipline han its origins in 
operational analysis techniques beginning in i'^Jorld /^ar II 
(/iilliams# 1^^78, o. 12). These techniques were tyoically 
aoplied to special tyoes of clear-cut# repetitive problems# 
such as those of systematic search and resource allocation. 



8 



Since the l^hO*Sf howevecf a more general technology 
has e^erqeo tor imcosina loaical structure on the reasoning 
that unrjerlies any sn^cific decision. fnis technoloay is 
decision analysis (Barclay et alw 1977^ o. iv). Since 
l^TOf there has been a najor effort by defense agencies to 
adoot this technology to their day“tO"“day decision ma»<inq. 
Many have found it a way to mai^e better/ more defens idle 
decisions. 

Decision analysis is a quantitative method which 
oermits the svstematic evaluation of the costs or benefits 
accruino from courses o*^ action that might be ta'<en in a 
decision oroblem (Barclay et al./ 1977/ o. vi). The method 
includes the iaentification of the alternative choices 
involved/ the assionment of values (costs/ oenefits) for 
oossi&le outcomes/ and the expression of the probability of 
those outcomes occurring. Once this is done/ the orobable 
aain or loss associated with each alternative can be 
rietermined bv systematically combining the probabilities and 



values. 

In addition to the primary role of decision analysis as 
a method for the loaical solution of com.plex decision 
problems/ it also has several additional advantages as well. 
The formal structure of decision analysis insLJres all the 
elements/ tneir relationships/ and their associated eights 
have been considered in the decision problem. The m.odel of 
the decision problem can serve an important role in 
facilitatina communications between those involved in the 
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decision orccess. ^Iso it is very easy to identify the 
areas of disaoreernentf their relative inportancef and if 
t*^ey actually have any niaterial in^oact on the indicated 
decision. Fin^ily^ when changes occur in the oroblem/ it is 
relatively easy to reenter the existinq oroblem structure to 
chanae values or to add or remove oroblem dimensions as 
reauirea (Barclay et al.# 1^77, o, vii). 

It should be emohasi^ed that in no sense does decision 
analysis reolace decision makers or the role of human 
judament in decision ^axina. Intuitive^ im.Dlicitf or 
judamental decision makina iSf after all/ the mainstay of 
the exoerienced decision maker and for good reason. 

Intuitive un- aided methods have typically served the 
oecision mak:er wellf and ne/she can reflect on some oersonal 
history of productive reliance on their developinq intuition 
(BrinxerSf 1^72# p, 1*^1). There is considerable evidence 
tc show that unaided decision makina is reasonable/ 

effective/ and reliable (^eterson and Beach/ 1967/ oo. 
However/ what riecision analysis does/ is orovicje an 
orderly and more easily understood structure that helos to 
aaoregate the wisdom of experts on the many tooics that may 
be needed to make a decision/ and then suoocrt the skilled 
decision maker by Providing him with sound techniques to 
supplement and ensure the internal consistency of his 
i u d g m e n t . 

Complex ipcision problems are often difficult to 



resol ve 



This occurs for a number of reasons 



Od t i on s are 



not always clea'^ly 



a e f 1 n e rj 



Any results which may be 



derive^ from the selection of a oarticular option may be 
hiahlv uncertain. Also/ it is often difficult to rJet^^rnine 
r^^lative orefemnces fon oossinle decision outcomes- 

'/hen problems such as these do occur/ the decision ma<er 
normally takes steos to structure the oroblem and reduce it 
to a more explicit form. This is exactly what decision 
analysis does. 

decision analysis builos uDon four basic elements whicn 
are inherent in any decision oroblem (Barclay et a1./ 1977/ 

o. 1). The use of these fotir elements allows for a smooth 
procedijre in the resolution of comolex decisions. The four 
elements are: 

1. A set of initial courses of action. You must 
have more tnan one alternative or there is no 
decision to be made. All possible alternatives 
snould he considereci without regard to 
plausibility at this ooint. 

The DOSS i ole conseauences of each initial act. 
These must be considered. What are the 

imoortant thinos that can happen that will make 
one act more valuable or worth more than another 
act? Relevant seauences of subsequent events 
and folio w-uD acts must be identified for each 
initial act. 

3. How attractive or unattractive is each 
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consequence 


o f 


each act? 


undesirahle 


1 s 


one ootco 


w n 1 c ^ m 1 d h t 


r »su 1 


t f r oT t 


decision. 






how likely i 


5 i t 


t n a t a 


result in 


each 


O t the 


o r 0 0 a b i 1 i tv 


or uncertainty 


crooaoi 1 i ty 


-) 

o 

3 


0 to 1 or 



oarticular act will 
conseauences. Inis 



These four eleTients^ as oescrilDed^ provide a way to 
organize/ auantify/ and trace the logical I'noli cations of 
the decision. Ihe nrirrary ooj active is to orovide a model 
of at least oart (all would he nest) of the decision. The 
use of the word model i^ this case means to reoresent the 
decision in a ouantifiable form. 

It is a central oreceot of aecision analysis that all 
relevant considerarions in a aecision can be represented 
fully in a decision diaoram ^Barclay et al«/ 1977/ o. 2). 
This decision diagram will show everything a decision ma^er 
feels is relevant to the oroblem in Question. A decision 
diagram consists essentially of a network of branches 
corresoonding to oossible sequences of acts and events/ 
fannino out from an oriain at the left to a time horizon at 
the riaht. Acta are available choices. Events are oossiole 
occurrences which are oartly or comoletely outside the 
decision maker’s control/ though the chance of one of them 
hr3pnenina may be influenced by acts which were carried out 
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earl i e r 



Tne decision iianra^r' orach icaHv rJistinguishes acts 
fron events. ^‘'ct for<s are reoresenteci by souares ancj event 
forces are reoresented by circles as sho'^n in Fioure 1. 

These act fori^s and event forks are then combined to 
fornn a decision tree. An examoie of a tree is shown in 
Figure 2. Tnis riecision situation involves a co^oany which 
nnust fiecibe wh^^th^r to bid on two oroJectSf ^ ann H. The 
decision on oroject A must he mane prior to the aecision on 
G. 

This decision diagram depicts all the oossirle acts and 
events and shows how these relate to each other in the 
decision situation. "Jhile the representation would help a 
decision maker to see at a glance his alternatives and 
identify those things that miaht affect any choice to be 
made# it does not yet answer the central Question: whicn 
choice should be made? That question can not be answered 
without considering tne value of the possible outcomes and 
the likelihood of occurrence of the events. In order to 
show how this is incorporated into the decision model# let's 
return to the oiidinq example. Let's say it has been 
calculated it will cost 510#00 to oreoare a bid for either 
oroject. If the hid on A is won# a oain of 550#000 will be 
realized. Project 6 will also return a gain of SS0#000 if 
the bid is won but due to overhead# will only return a gain 
of B20#000 if bio A is also won. The decision tree now 

The numbers shown at the ric^ht side of 
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looks like Figure 5 



OPTION A 





FIGURE 1: ACT AND EVENT FORKS 



1 a 



WIN B 




FIGURE 2: DECISION TREE STRUCTURE 
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WIN B 




FIGURE 3: DECISION TREE WITH VALUES ADDED 
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the diaare'^ are tne oath values 



These values are the sum 



of the costs and gains alom each oath, 

^.Cv^/ to ccToleto the oiaoreTi/ we neeo to oort^ay tne 
degree of uncertainty about the events- we cannot controls 
soecifically whether a oiH is won or lost. Based uoon orior 
exoerience/ it is ju daeo that the bid on oroject A is 
equally likely to be won or lost (orobability ,5 win# .5 
lose). If A is won# the oronebility of winning the bid on 6 
is reduced to -d. However# if A is lost# the chances of 
w inn inn R increase to ,7# and if A is not oisd on# the 
chances for 8 are -o to win, Adoina these values leaves our 
diaqram looking like Figure 

This aiaaram should now be a virtually comolete 
translation or moael of the rercection of the decision maker 
of the decision nrohle^. All that is left now is to 
determine which is the best solution. The method to do this 
involves tne calculation of weighted values# often called 
expected values# for each decision ontion. This technique 
is the simolest and statistically most straightforward 
method that can oe used. Care must be taken however# since 
frequently there are decision circumstances wherein an 
expected value solution to a decision problem may not be an 
optimal one. In these cases# alternative means of treating 
value are reouirea and will he discussed later, 

Ceterminina a value for each act in a decision is done 
by a procedure called folding back the decision diagram. 
Values are substituted for each act and event fork beginning 



WIN B 




FIGURE 4: DECISION TREE WITH PROBABILITIES ADDED 



+ 50K 



+ 30K 



+ 40K 
+ 30K 



-20K 

-lOK 
+ 40K 



-lOK 

0 




at the riqhr-han<i side of tHe oiaarann. At eacH event fork, 
the exoecte^i value of that event is calculaten by 
nultiolvino the value of eacn oossible outcome by the 
orooabilitv of ^he occurrence of that outcome- At each act 
fork, the act with the hiahest value is the one which should 
aive the oest decision. The crocess of substitution is 
continued until the initial act fork is reached. Figure 5 
deoicts this nrocedurp, Tj^orefore the decision t. a<er would 



b i d 


on 


o r o j e c t 


A , then if 


he ^on the hid. 


not bid on Project 


p 


I f 


he lost 


the t/ ' d OP 


A, he would Did 


on project B. 



Thtjs far in the discussion of decision analysis, the 
values of tne gains or losses were assigned in terms of only 
one meastjre, mor'i^=v« T outcomes were then comoared, using 
the exoected value, very easily since it is easy for us to 
relate to money as a value. However, in most real life 
situations, esoeciallv those of the military, gains and 
losses are not m^f^asured in money, rather in terrain, 
eduigm ent, caoabilities, and oy en lives (/nil i a ms, 1^78, o. 

These value dimensions are Qualitative rather than 
quantitative and are very difficult to measure. To further 
cloud the issue, each individual has his own oersonal 
subjective values for each of these. 

In addition, very seldom are these value dimensions or 
attributes given egual weight in a decision. For examole, 
consider the selection of a radio set from several comoetinq 
models. Cost mav oe an imoortcant consideration, but so are 
weight, range, oortability, and reliability. Just how 
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WIN B 



+ 50K 



.4 



BID ON B 




EV(7) 
EV (8) 
EV (6) 
EV (4) 
EV(5) 
EV (3) 
EV(2) 
EV (1) 



( .4) (50K) + ( . 6) (30K) = 38K 

( . 7) (30K) + ( . 3) (-20K) = 15K 

( .6) (40K)+( .4) (-10K) = 20K 

LARGER OF 38K OR 40K = 40K 
LARGER OF 15K OR -20K = 15K 
LARGER OF 20K OR 0 = 20K 
( . 5) (40K) + ( . 5) (15K) = 27. 5K 

LARGER OF 20K OR 27. 5K = 27. 5K 



FIGURE 5; DECISION TREE WITH EXPECTED VALUES 
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iToortant each is requires aeterTi nation, /'hat is imoortant 
in t^^e oecision is which of these attrioutes is more 
imoortant ^nan the otners and how is this difference scaled. 
Then now do the remain i no attrioutes fit on this scale. For 
instance^ are cost and reliability eaually important? If 
notf what exactly is the difference and how does the 
decision ma'or make trade-offs between the two. 
Oot-erm. i net ions of this sort must Oe made for all attributes 
and somehow our model must ce anle to comoare across these 
many attrioutes and aoorenat^ the results to indicate the 
best solution. 

Problems of this nature have sourred the cjevelcoment of 
multi-attribute utility models (/iilliams# 1978^ o. 2^). 
These me a els nelc to crevide a relative rankina for 
attributes and also orovide a common auide for aaqreqating 
the measures into a sinole index of worth. This orocess 
involves the assionment of a measure of utility or merit to 
the attributes. 

Utility can be described as a subjective measure of 
" liking ” (Farclav et al.# 1977, o. 27). It is a personal 
value reflecting how you subjectively value something. The 
aoplication of utility as a measure for the value of an 
attribute or alternative was first proposed by Von Neuman 
and Morgenstern. They suoaested that eacn individual has a 
measurable nrpferencp among various choices available. The 
preference they calleci utility and is measured in arbitrary 
units called utiles (Von Neuman and ''’oraenstern, 19^4, p. 
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??)• utility is baseU on the conceot that in decision 
akin Of a ce^son v^ill choose that alternative /^hich 
mayirri^es nis or her «^xoected utility . 

'‘''Mjlti-attrioute utility models have been used 

extensively in the systems acauisition role (i^villiams# 1978# 
o. 23). The Droceoures encourage discussion among the 
decision 'r>axer and his stafT, r'io longer can cost be a 
little more imoortant than reliability. It is now# oerhaos# 
ten utiles more imoortant. Once the values are assigned to 
the model# ancj a best decision decioed uoon# the model no a 
allows you to vary the relative weights to see which ones 
have the most imoact on the proposed solution. This is 
called a sensitivity analysis and helos decision makers when 
they are uncertain about the accuracy of their information. 

Decision analysis aoolies Bayesian techniques to olace 
a value on information that reduces uncertainty (Keen and 
Scott Morton# 1976# c. Bayesian techniques provide 

formal methodologies for analyzing the imolications of a 
decision matter's subjective judgment of probabilities# 
updating these assessments as additional information is 
obtained. This additional information can result in one of 
two events: either the uncertainty is completely removed or 

the additional information allows the decision maker to 
revise the initial Cor i or) assessment of the probabilities 
(Keen and Scott ^^orton# 1978, o. ^6). 

As stated earlier# the use of decision analysis does 
not reolace the decision maker hut ados a new dimension to 
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his aecision rrakina caoabiHties. It is also assum^O the 
use of tnese manual tec^ninues could slow the decision 
n^akino oroaress consioerablvf especially if a fear of 
Tathe^atics exists. ^orri.inat*ply/ what ^as oeen develooed 
are decision aids which greatly ease the complexity 

i n V 0 1 V e cj • 

A decision aid is a human-system interface designed for 
the specific ouroose of supoortina and enhancing a decision 
maker in his decision ma^'ina role (Keen and Scott '^‘'orton^ 
n. 58). It is a tool for use by the decision maker. 
Decision aids are normally stereo on conouters as th<= use of 
the computer reduces all needs for calculations by the 
decision maKer and helos to soeed the decision process. 

The general availability of low cost/ high capacity/ 

fast information orocessina technoloay has enabled man to 
extend and increase his intellectual capacity. The effect 
of this has been that man can now deal more effectively with 
complex matters on his own comolex terms as aoainst the more 
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terms 
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man 


(Br i nkers » 


1972, D. 


5) . 


The focus 


is not 


on the computer 


i t- se 1 f , 


but on 


the 


technol coy 


for 


dealing with 


the 



information that is available and the scientific methods 
which are in development and use. 

The use of decision aids normally places increased 
demands upon human creativity and judgment rather than 
relieving the decision maker of the neeci to exercise them as 
might be expected. Tnis is because the availability and use 
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of oecisiof' which serv® as an extension of tne decision 



-naker's intellect provide new and interestinq oddo r t un i t i e s 
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T I . II3F OF t]r>3 IT H I M FHF 



CoTimand, control end conrrun i c a t i on s (C3) can be defined 
as a orocess which orovides the commander (or decision 
maker) with a means of receiving information, making 

decisions based on this information, and then imolementing 
and monitoring the decision in order to achieve his or her 
mission r-'oose, l<^dO). ^ C3 system includes the data 

(or intel 1 ioenc») orocess, t he dec i s i on ma k i ng 
process, and the oceratinq forces (or weaoons) required to 
achieve the mission. ^ C3 svstem would look somewhat like 
the figure shown belo-'. 




FIGURE 6: C3 SYSTEM 
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This figure sho'^s tnat data is aatnereo# oossioly 
through the use of early >,arninq raoars^ ohotograDhy, or 
reoorts fro'^ vit-hin, ano sent to a cofr^mancj center. Once 
within the connnancj center, tae oata is analyzed ano a ae* 
cision is nnade reoaroing the use of weanons or units. These 
decisions are then transrritted to the resoective units for 
implementation. Peouirements for data or information could 
originate at the command center or it could he soontane- 
ouslv Generated as in the case of a missile attack. 

Unfortunately^ t^'e majority (if net all) of the 
resources soent in thc^ imorovement of these systems are 
expenOeo in either oroviaing cetter data aatherina (such as 
better radars)^ failsafe communications# or biqaer and 
better weaoens systems. These are all important aspects and 
shoulci he improved, put this is also indicative of the lack 
of aopreciation for what must be done ^ith the data when it 
is delivereo to the command center. Little thouaht is given 
to row the decision maker processes the the data-in order to 
make effect- ive cecisions and forecasts. 

The amount of data available today to decision makers 
is m i ndbooG 1 i nq . R a 1 i s t i c a 1 1 v # if methods for cognitive 
information process ina are not improved at the same rate as 
our communications can provide it# much# if not all of our 
sophisticated communications technoloay will go under 
utilized. /vhat is needed is more research aimed at 
imeroving our information management# decision making, and 
forecasting utilizing th<^ applications available through the 
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Decision t a kino lies at the heart of the C3 orocess 
(Ancirioler o. 1?). The entire C5 system focuses on 
orovinina a means to transmit all reauir^o data to the 
decision maker and tren a means to transmit the decision to 
the reauired oarties. Therefore# C3 svstems which do not 
contain some sort of decision making sucoort should be 
chall^ngeo because they are incomplete. 

^^‘ost decision makers will begin with a neeri for some 
bac^arouno information on the oroblem at hand. Has there 
been a problem like tnis oefore? Hpw have the people 
involved react in similar situations? Are there any 
oeoole available who have experience in this area? These 
are all examples of the tvoes of Questions which will need 
to be answe red • 



Next# the decision n^aker will want all of the latest 
facts surrounaino the Problem, i"iho is available to move 
into the area? ‘■^ow Iona will it take? .’'Iho is there 
already? '.'hat forces does the ooposinc sine have available? 
'A hat is happening now? These and many more are the 
Questions surroundino the current situation. 

Mow# the decision maker will want to know what the 
oDtions are and what the adversaries options are. In 
addition# will be the need to know of any activities# either 
in process or Pend i no# which mav nave an impact on the 
decision. '^^ext is the assessment of the probabilities of 
the Possible outcomes of these activities. The criteria 
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wi^ich /rfill be used in c^nelyzing tne al^e^nat^vf^s must be 
listed ang rei;^tive imcorfance weiohts for these determined. 
Finallv the decision 'T^ai<'er evaluates the alternatives and 
selects a course of action to be followed. 

How does t^^e decision maker do all this/ esoecially 
•when the imoacts of a wrong decision can be so costly? It 
certainly can not be donp alone. The oecision maker must 
relv on the aid of key staff n^embers who have certain 
exoertises in soecific areas. But even with comoetent 
staffs/ the task is often too areat. Como uteri zed decision 
aids may be the only answer. 

There are several reasons why comouterized decision 
aids would oe useful in the C3 decision making orocess 
(Allen et al./ 1976/ oo. ). These reasons are: 



1. It is an effective tool for imorovino the 
dialodue among staff officers who are workina 
tooett^er on a oarticular nroblem. It would be 
heloful in eliminating misunderstandinos. 

The process of using a decision aid gives staff 
officers the ooportunitv to include what they 
considers important in the analysis. Then they 
can see how these factors actually imoact on the 
solution. 
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Critical areas can now oe ^oc'jseo on. ^11 too 
often^ too Tuch tinne is waste ri in discussing or 
arouino wnat i^av seenn to he imoortant points^ 
but turn out to be rather insignificant in the 
final solution. 



5. 



Tire T'av no m ba in your favor ratner than your 
occonent. *'^'ost oeoole do not react well when 
under tirre constraints. Peocle tend to over 
react to uni^ioortant events ann make Quick and 
hasty judonnents* Decision aids enable and force 
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There are nuTerous exa'^ples of C3 problems where 
comouteri^ed decision aids would be useful and the following 
are so^ie exanoles: 



1. Decision aids can and are being used to monitor 
early warnina systems for any indications of 
imoending crisis. It is important to not allow 
events to con^'rol uSf but rather to possibly 
have us shan^ the events (Andriole^ 1^80/ o. 
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conc^rrerj analysis of 
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?. A 

oifferenr evacuation 
was consider inn 
a dev e 1 oo i nn 
necessary to 
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oostures that a connancier 
face of uncertainty about 
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U.S, nationals frorn a 
o. ^7). 

process that 
another exarnole 
The decision aid 
"what if" qame. 
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crisis 



<=^vacuate 

r y f ^ 1 1 1 a n s / 1^78/ 



continaency olanninq 
rai<e3 olace is 
1^78. D. 28). 
to easily p1av the 



Trainincj is another exanole. 6y allowinq staff 
officers the chance to use the aids based on 
past crises/ they becone familiar with current 
plans ana procedures and/ if realistically used/ 
oecome used to ooeratina in a crisis environment 
(Allen et al*/ 1^76/ p. S). 



5. Decision aids may be used for establishinq 
policy. Options can be evaluated and tested 
prior to decidinq on firm policy. 

6. The desivon of new C3 systems would be another 
example. '/ihenever new systems are aesianed/ 
there is always a tradeoff of caoabilities 
because of insufficient funding. These aids 
would help fjetermine the safest cuts in 
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IFI. EXISTING C3 DECISION AIDS 



This chaoter aescribes several C3 c om ou t e r -ba sea 
information^ becision/ and forecasting systems desianed to 
1in< the human user with the information so critical to 
efficient information manaaement. 

There oresentlv exists much too little emohasis uoon 
tne content and esoeciallv the form of the information and 
data that flow thro u ah 03 svstems (Andriole^ 1^80^ o. 3). 
The Questions of how the information should aooear to the 
user, how the information should be storea, retrieved, and 
manioulated, ano how the ever increasino amounts of 
information can be comorehended by the user have not been 
adeouately dealt with uo to this point- The three 
computer-based information systems described below were 
developed with these Questions in mind. Each is aimed at 
helpino th« C3 system users deal with the enormous amount of 
information for which they are responsible. The systems 
were developed with the human user in mind and thus exploit 
and supplement existing human information processing 
caoabi 1 ities. 

Soatial Data Base '’^’anagement - The use of normal data 
base manaaement systems, from the perspective of the 
occasional user, are difficult and inefficient (Andriole# 
1^80, n, 3). Recently# research has produced a prototype 
spatial data management system which enables users to 
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hierarcHically store# rptrieve# ano manioulate data through 
tne use of scat i el controls (Andriole# 19^0/ p. a)* These 
soatial controls allow the user access to data v>ithout the 
need or us^ of the conventional keyboard. Instead# the 
soatial controls oermit access to data using numerous 
navaoational aids# such as color# location (in an electronic 
workspace projected on a large screen display)# touch# and 
sound. Specifically# the user may store and retrieve the 
data accordino to where he or she finds it most easy to 
access. 

The soatial data manaaem. ent system was develooed in 
accordance with the way humans normally store and retrieve 
information. For examoler in a normal office environment# 
oeoole store information in familiar claces according to 
freduencv of use# imoortance# shaoe# size# ana so forth. 
The orotctvoe soatial data base manaoement system uses these 
osycholoqical oredisoositions by allowing the user to create 
nis or her own osvcholooical worksoace and oroject it onto a 
lame disolav. This display then is interactive and caoable 
of storing all types of data to include numeric# 
ohotooraohic# and even audio. Tt is also easy to update. 
The user is then able to manipulate and retrieve data by 
movino through the workspace with joysticks or touch 
sensitive disolav panels. Figure 7 shows the hierarchical 
data organization principle* Figure ^ shows a user 
positioned in * rant of a soatial data base management 
system. Figure 9 shows the joystick and touch panel mounted 
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FIGURE 7; 



HIERARCHICAL DATA STRUCTURE 
(Andriole, 1980, P. 5) 



3 ^ 




FIGURE 8: THE PROTOTYPE SPATIAL DATA 

MANAGEMENT SYSTEM (Andriole, 
1980, P. 5) 
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FIGURE 9 : 



DATA ACCESS CONTROLS 
(Andriole, 1980, P. 6) 
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in the arr of a user’s chair. Oata is accesses by touching 
a SPeciHc point in thp ^A/ori^'soace or the touch-sensitive pad 
and/or ov Bovina the joystick forward or oackwara (to ascend 
or Oesceno into t e hierarchically oraanized data base). 

A ci active C3 Information Selection - In resoonse to the 
ever increasing information requiren)ents olaced upon 
intelligence of*^icers responsible for analyzing ana routing 
information^ the Adaptive Information Selector (AIS) was 
develooeo ('^"adni et al.r 1979, p. 1-3). During this 
development orocesSf it as determined that ”new C3 
techniques are requiren to control information flow so as to 
best match svsterr capability with human characteristics in 
the man-comPuter interaction”. 

The AIS process is as follows. The system user is 
calibrated during a training ohase to determine how and why 
some messages (pieces of information) are selected for 
further analysis and some are rejectee. This is performed 
for eacn inoividual- user such that he or she has a 
personalized inforT>ation nranagement sort i no ^ routing^ and 
Queuing system. 

What happens# in effect# is the computer monitors the 
user and then internalizes how the user selects and rejects 
information. Once the training program is accomplished# the 
computer emulates the user and automatically selects# 
routes# and queues the information. Thus the AIS may be 
viewed as an information management assistant capable of 
relievino its superior from some of the burdens of 
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infoPTr^^tion overloari. 

The functions a^ri elements of an ^IS are presented 
araohic^Hv in Fiour^^ 10. 



Training Channel 




Manual Channel 



FIGURE 10: ADAPTIVE INFORMATION SELECTION 

SYSTEM CONCEPT (Andriole, 1980, 
P . 8 ) 



The AIS has been developed for use where C3 information 
is already in an electronic form. The AIS is designed to 
suDCort areas which are already comolicated wit^ too much 
incominQ information for the human user to handle. The AIS 
is merely a computer orooram which provides helo to the C3 
system user. 

Ultra-Rapid Rea dina • Even if systems such as soatial 






data base 



rranaoe-nent and AIS Cr?n auic<lv orovide access to 



needed intorTatioo/ the user T^ust still reao ann corr>creKena 
tre information oefore further action can be contemolated. 
Suoseou^ntl// research has o^en performed to irrorove the 
soeed with which users can reao and comorehend (Andriole/ 
1980/ oo* 9-11). 

A technique known as raoid serial visual oresentation 
(9SVP) has oeen oevelooeo and conouteri zed for use in C3 
svstens. The text is oresented one ^ord at a tine on a 
cathode rav tube (C^T) in raoio succession as Finure 11 
suggest s . 



THE 



1 



ULTRA I 



RAPID 
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RE ADER ~) 
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TEXT 



ONE 



WORD 



AT 



TIME 



FIGURE 11: ULTRA-RAPID READING 

(Andriole, 1980, P. 10) 
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^esearc^^ 



in 'He 3 ted that sinale ^SVP 



sent ences can 



be rea^ an^j accurately recalled when shown at a rate as hioh 
as \ d woras opr seconif which is twice as fast as oeople 
normally r^ad (hotter et al./ o. 1), 
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oictures or symbols for woros has increased comorehension. 
However^ as with AIS/ the text must oe in electronic form to 
be rea.d ultra'-raoidly. 

The orecedina three research desians were directed 
tov/ard imorovina the information flow to the decision maker. 
The next three examples are computerized decision aids which 
are desiqned to be used orimarily during crisis situations. 
Each r:>ermits the user to select and retrieve history on oast 
crisis situations. The best wav to describe each is by 
providinq a samole of the types of Queries which are 
allowed. The first/ as described in Figure 12 / enables a 
user to retrieve and prescribe crisis actions (or responses) 
and objectives. The second aid/ as shown in Fiqure 13 / 
enables a user to focus on the manaaement croblems which 
occurred durinc oast crises. The third aid enables a user 



ao 



to exarr^ine 



307 crises involvino the U.S 



h e t e e n 



19^6 a n ( j 



1‘^7b, as f-iaur? I -l snows. 

jhose ai'is were develooe(i direct response to 

reouests fro-n crisis managers for some tyoe of on-line 

analytical assistance (Andriole^ May-. June 1^79, d. IS). 
Crisis managers are now able to assemble^ sort» and analyze 
crisis manaoeTtent data on a real time basis. Preliminary 
oerformance tests of these aids suoaest they might increase 

ranoe of oction o‘=neration and evaluation durina crisis 
ny 100% and reduce oast search case time by 50%. 



THIS SECTION IS DESIGNED TO ASSIST DEPARTMENT OF DEFENSE PERSONNEL IN 
EVALUATING PROPOSED COURSES OF ACTION AND SETS OF U.S. OBJECTIVES BASED 
ON DATA FROM 101 CRISES INVOLVING THE UNITED STATES BETWEEN 1956-1378. 

THE USER IS GIVEN THREE LEVELS OF ANALYTIC ASSISTANCE: 

1. THE CAPACITY TO SEARCH FOR HISTORICAL CASES WITH 
SETS OF USER-SPECIFIED U.S. ACTIONS OR OBJECTIVES. 



2. THE CAPACITY TO IDENTIFY. ACROSS ALL CASES. 
THOSE ACTIONS THAT HAVE HISTORICALLY BEEN 
MOST STRONGLY ASSOCIATED WITH EACH OBJECTIVE 
SELECTED BY THE PROGRAM-USER; 



3. THE CAPACITY TO IDENTIFY, ACROSS ALL CASES, 

THOSE ACTIONS THAT HAVE HISTORICALLY BEEN MOST 
COMMONLY ASSOCIATED WITH SETS OF U.S. OBJECTIVES 
SELECTED BY THE PROGRAM-USER. 

WOULD YOU UKE TO SEE A SUMMARY OF ANOTHER SYSTEM SECTION? 

PRESS Y’ OR 'N' AND 'RETURN'. 

Y 

ENTER 'I', 'II', OR 'III'. 

II 



FIGURE 12: 



CRISIS MANAGEMENT DECISION AID 
(Andriole, 1979, P. 16) 
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THIS SECTION PERMITS A DETAILED EXAMINATION OF MANAGEMENT PROBLEMS 
ENCOUNTERED IN 41 SELECTED CASES (1956-1976). 

MAJOR PROBLEM CATEGORIES ARE: 

1. SYSTEM-RELATED DELAYS IN DECISION-MAKING 

2. SYSTEM/PROCEDURAL CONSTRAINTS ON ACTIONS 

3. LEGAL ISSUES INVOLVED 

4. RESOURCES INADEQUATE FOR OECISION-MAKING/ACTION 

5. INTELLIGENCE FAILURES AT DECISION-MAKING LEVEL 

6. EMOTIONAL/IOEOLOGICAL ISSUES INVOLVED IN DECISIONS 

7. INTERPERSONAL FACTORS IN DECISION-MAKING 

8. PROLONGED CRISIS PROBLEMS 

9. PROBLEMS IN SELECTING ACTION PERSONNEL 

10. CONSTRAINTS ON OPERATIONS 

11. PHYSIOLOGICAL PROBLEMS FOR OPERATING FORCES 

12. INFORMATION FAILURES BY OPERATING FORCES 

13. FAILURES IN TAKING APPROPRIATE/TIMELY ACTION 

14. FORST AT PROBLEMS 

15. PROBLEMS IN THE OPERATING ENVIRONMENT 

16. GENERAL PROBLEMS IN CRISIS PLANNING 

17. GENERAL PROBLEMS IN CRISIS HANDLING 

18. GENERAL PROBLEMS IN CRISIS TIMING 

WOULD YOU UKE TO SEE A SUMMARY OF ANOTHER SYSTEM SECTION? 

PRESS 'Y' OR 'N' AND 'RETURN'. 



Y 

ENTER M', 'ir OR 'III'. 
Ill 



FIGURE 13: 



CRISIS MANAGEMENT PROBLEM ANALYZER 
(Andriole, 1979, P. 17) 



THIS SECTION PROVIDES ACCESS TO INFORMATION ON 307 CRISES (1946-1S76). 

THE DATA MAY BE SEARCHED FOR CASES MATCHING THE USER'S SPECIRCATIONS. 
OR A FUa DESCRIPTION MAY BE PRINTED FOR ANY SELECTED CASE, 

CATEGORIES OF INFORMATION CODED FOR EACH CRISIS ARE: 

• YEAR AND BRIEF DESCRIPTION 

• LOCATION OF CRISIS 

• "nature or pre-crisis ACTIVITY 

• DURATION OF PRE-CRISIS PERIOD 

• SCOPE OF CRISIS (DOMESTIC OR INTERNATIONAL! 

• NATURE OF CRISIS (MILITARY,POUTICAUBOTH) 

• CRISIS DURATION 

• TIMING OF CRISIS RESOLUTION 

• CRISIS OUTCOME 

• ANTICIPATION OF CRISIS 

• DEGREE OF THREATTOU.S, INTERESTS 

• TIMING OF THREAT DEVELOPMENT 

• TIME AVAILABLE FOR DECISION 

• SIZES OF PARTICIPANTS/DEGREE OF U.S. INTERESTS 

• U.S. RESPONSE/PARTICIPATION 

• U.S. OBJECTIVES 

• NUCLEAR/NON-NUCLEAR IMPLICATIONS 

WOULD YOU UKE TO SEE A SUMMARY OF ANOTHER SYSTEM SECTION? 

PRESS 'Y' OR 'N' AND 'RETURN'. 

N 



FIGURE 14 : 



CRISIS DESCRIPTOR 
(Andriole, 1979, P 



17) 



Ourinq crisis sit'uat'ions/ decision -makers strive to 
react swi^tlvr decide »*/iselv^ and coTinunicate accurately. 
Inis requires close coordination i^itH their staffs and the 
ability to overcome certain otnstacles such as pressures from 
time constraints/ amhiauity of goals/ and the monopolization 
of time with information collection. Some of the problems 
of information collection nave been solved or at least aided 
by the iis#=^ of one or 't^ore of the oreviously discussed aids, 
/‘ihat the decision '^aker now needs are effective (decision 
strategies that impose rigor and provide a logical/ 
structural framework to assist them in the Process of 
choosinc an optimal decision alternative in the face of 
voluminous and often inconclusive evidence. 

Operations and Intelligence (OPI^iT) is a decision tool 
that orovides )ust such a framework for deliberation/ 
reasoning/ and analysis (Amey et al./ 1^79, d. 3). OPINT 
is an on-line/ interactive/ real-time model which aids 
decision makers by prescribing a s t r a i qh t f o rwa r c normative 
orocedure for organizing anj analyzing difficult decision 
oroblems. These problems may involve both uncertainty about 
the outcome of future events and perplexity about the 
complex value trade-offs involved in the choice of a course 
of action. 

G^^INT IS a decision-analytic based/ computer assisted 
decision aid. Its crimarv objective is to provide decision 
makers a orocecural framewor</ or decision template/ that 
insures their ultimate decision choice is a coherent one. A 
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coherent cnoice is one which is consistent with their own 
value structures and beliefs about the relative lii<e1ihoods 
o^ future ev^^nrs tnat will in^oact the decision outcorre (Anney 
et alw 197^^ o. 3). 

The fundaiiental oroduct of OPINT is a comouter-stored 
conceotual reoresen t a t i on / or decision models of the 
decision oroblen at hand. /'ihereas decision analysis 
orovides the theoretical background and procedural guidance/ 
the OPI'iT decision model provides the soecific 
method oloaical tool for orocessino information and 
evaluatina the various decision alternatives ooen to the 
decision maker. An in-deoth exolanation of how OPIMT aids 
in the (decision process is discussed in Chacter Four. 

Evaluation (EVAL)^ another on^-line/ interactive/ 
real-time model/ incorporates another decision strategy 
which orovides the aecision maker with a normative procedure 
for analyzing difficult decision problems. It is especially 
desicned to evaluate end compare possible options using a 
multi - attribute utility analysis technioue. EVAL, is best 
applied to orocurement decisions wherein systems are 
compared/ but is appropriate for a diverse set of decision 
analytic problems (Decisions and Designs/ Inc./ 1^77f po. 
1-5) . 

The use of EV^L is broken into two phases. First/ the 
decision maker creates the structures of a hierarchical 
multi -attribute evaluation model using the EVAL Structure 
orogram. Then/ this model is evaluated and processed using 



as 



the EVAL nro^ran 



In order to utilize the EV^L Structure orooram^ the 
oroolem ntjst be oeco^^ooseo into a hierarchical structure 
which reflects th<^ loo i cal interrelationshio of all the 
factors involved. This structure is then entered and the 
model is createcj. Now under the direction of the EVAL 
oroaram, the criterion used for evaluating the alternatives 
are entered. flnce entered# their resoective weiahts of 
iToortance are entered. iNext^ the relative score which each 
alternative achieved for each criteria is entered. For 
examole# consider three radio sets which are being evaluated 
for selection before orocu remen t (Al, A2# and A3), The 
criterion of evaluation are cost# range# oortability# and 
reliability. Each of these criteria are weighted 30?^/ 20%^ 
10%# and ^0% resoectively. N'ext# the three radio sets are 
evaluated accoraina to each criteria using an arbitrary 
utility scale. Cost would show scores of 50 for set Al, aO 
for set A2# and bO for set A3. This means A3 is considered 
to have the best cost. This process is repeated for each 
criteria. It is also normal for the criteria to have 
sub~criteria which would also reauire the same procedure. 

Mext/ EVAL gives the evaluation scores for each of the 
alternative systems resulting from the agareqations 
orescribed in the model structure. Mow the decision maker 
can examine the effect of the criterion weights throuah a 
sensitivity analysis. This will quickly show him which 
criteria are the critical ones and take action to insure the 
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data surrounnina th^^t criteria is as accurate as oossible* 
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Another decision aid has been develooea to assist 
qrouDS in deciding how to deal with imoortant issues or 
oroble'ns. Its ouroose is the i.TioroveTent of decision making 
by continuous oarticicant interaction with the comouter 
during the decision making rrocess (Leal et al.r 1978^ o. 
l-l), ■'^embers of the arouo are allowed to input their 
respective estimates of the occurrence of specific events 
and the in values (weights) of the importance of specific 
decision outcomes. The decision aid then compares the inout 
and informs tne aroLJO members of any disagreements between 
them. but most importantly# it focuses the discussion only 
on the disagreements w^ich are important to the stated goal. 
Fne differences which are not important are shown to be just 
that. Finally# it sijogests to the group precisely how their 
disagreements mioht be resolved. This enables groups to 
move much more Quickly toward decisions. As a by-product# 
profiles are develooed of the participants which show who 
has expertise in soecial areas as well as who is a 
risk-avoider or risk-taker (Leal et a1.# 1978# pp. 1-3 to 
1*10). The function of ^ne aid is shown in Fiaure 15. 
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FIGURE 15: THE GROUP DECISION AID PROCESS 

(Andriole, 1979, P. 17) 



The decision qrouo is corroosed of the particioants (de“ 
cision Ta<ers)» an i n t e r Ted i a f 0 r , ann a director. 
The interTediator's oriTorv function is to facil- 
itate co-^Tun i cat i on between the qrouo and the comouter. 
This is done bv translation snoken requests into conout- 
er inputs. These inputs could be lists of events, alterna- 
tive actions, requests for displays, or modifications 
to any orevious inputs. The director helos with the outout 
by oresentina and explainino it to the participants. 
The director also focuses the group's activities and insures 
the incuts are appropriate. ^s a result, virtually no 
participant trainino is needed to operate the system, and a 



as 



Group can oeqin cn its decision proolem al^^ost 
i.Timenir3te1y after it is convened. 

-nether conputerizpG aid/ the torly /earning ana 
‘’^onitorino Syste*^ (F/''AMS)/ has been aevelooed to aid the 
Indications ano l^iarnina (li/i) community. orovides 
them with an interactiv<=^ computer based system of oolitical/ 
military/ and domestic indicators for daily monitoring of 
international anci intra national affairs (Daly/ 1^78/ o. 
^). The system nas the caoability to store Quantitative 
international oolitical data from t^6o for most countries of 



the world 


and 


can 


therefore 


oe usea to 


do retrospective 


analysis as 


M e ] 


11 as 


c u r r en t / 


daily I 


. The data 


1 s 


aag r eqa t ed 


by 


TOn t H » 


n u a r t e r 


/ and year 


Ei-<A'.1S has 


the 


c a n a b i 1 i t y 


t 0 


t rac Ic 


single 


countries/ 


combinat ions 


0 f 



country-pairS/ regions or the entire world with these 
Quantitative indicators/ or soecific reoions may be 

soecified (Oaly^ 1^78/ d. o). present/ there is some 

doubt as to whetner E-^iA'^S will ever be fieldec:: because its 
operation is extremely slow. But the concept is a good one 
and further research may prove beneficial. 

The E/jAMS is coT.orised of the following comoonents 
(Andriole/ 19 3<)/ on. 

1. General Scans - A aeneral scan is an aggregation 
of countries by some criteria/ e.g./ a scan 
defined bv -geographic region would aggregate 
countries by orouos such as Latin America or 
Middle Eastern. A general scan allows an 



a n a 1 V s 



fo ]ooi< at several countries taken 



together rather tnan lookina separately at rrany 
country-pairs. If the indications for the yrouc 
as a wnol-^ suqaest unusually hioh activity/ 
tension/ or uncertainty/ the analyst can drop to 
the c oun t r y -by -c cun t r y level or even track the 
recent activity of a s inale country to determine 
tne source of the disturbanc‘=^ or the reqional 
level. General scans thus ma<e the system much 
more efficient and less time consuminq for the 
anal y s t . 

2. Quantitative/ Political/ Military/ Economic/ and 
Domestic Indicators - Quantitative inoicators 
for crisis warninq include fore i an/ domestic/ 
and international political/ military/ and 
economic factors. I'he indicators are both 
dynamic (events-baseo) and static (attribute 
based) . 

3. Multi Method Forecastina Capability - The 
unified multi-method forecasting capability 
reauires the system to Generate different kinds 
of forecasts or warn inns via different methods 
for different events and conditions.. Sucn 
options have been designed with reference to 
objects/ aoals/ and methods of forecasting. 
Experimentation is still beina performed in this 
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a re^ 



A d d i t i on a 1 


1 research 


is currently 


oe i nq 


Performed 


t 0 


aesian an aid 


« n i c h will 


focus on Sov 


let c r i 


s i s 


b e h a v i 


0 r 


since '^0 r 1 d 


N d r II. 


This aid will 


include 


the 


f o 1 1 o w i 


nq 



tasks: 

1* Develoo^ent of an inventory of Soviet crisis 
manaoe'^ent avion/ 

?. Collection o^ oata on the characteristics of 
thesp crises to show the nature of Soviet 
military crises. 



A more 


intensive 


anal 


ysis and codinq 


o f 


key 


S 0 V i e t 


crises 


t 0 


i ci e n t i f y the 


c r i 


s i s 


environments that 


ma V 


affect 


the occur renc e 


0 f 


crisis 


manaaement 


D rob 1 ems 


encountered 


by 


the 


S o V i e t 


Union/ and 


variables 


d e s c r i b i n o 


c r 1 


S 1 s 


oojectivesr actions/ 


and results for the 


CO 

o 

< 


i e t 


Union. 














Statistical analyses 


of the 


characteri st ics 


o f 


Soviet 


crisis 


manaoemen t 


ooerat i ons 


/ 


the 



environment in which thev have taken olace# and 
the crisis manaoement oroolems encountered by 
tne Soviet Union. 



5. Comoarison of these statistical analyses with 
earlier analvses of the same factors for the 
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u.s 



A]] these decision aic-is have been aovertised as 
beina valuable to incision ^a^ina. Mone have been 
exoeri mentally evaluated vvith kncwledqeable subjects in a 
decision ma^inq environment. A baseline of such evaluations 
is imoortant to establish in order to orovide a useful tool 
and better yet# to orovide for much needed technology 
transfer and real A/orldf real time feedback on the value 
(utility) of the tecnnoloriy. 

OPPlT was selectpd as an acorooriate candidate for 
exoeri mentation for a number of reasons. First/ it is 
readily available as a oackaoe at tne ^^aval Postoraauate 
School. Second/ all of the equipment required for its 
oceration already exists at the School. Finally/ due to 
both time and monetary constraints/ the procurement of other 
aids uas unfeasible. 
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W. FUNCT OFSCr^IPT lOfM OF OPINT 



OPIImT is a aecision*analytic baseo^ comnuter-assisted 
decision strateov (A-nev et* alwl^Zb/ n. 4). It was 
desianed an^d built by Decisions and Desions Inc.# of clean# 
Viroinia# under contract to the Defense Advanced Research 
Projects ^cencv fDARPA). It was written in the APL 
oroora^iriino lannuace for the 510 0 connnuter* In 
addition/ has recently been rewritten in FORTRA-'vl UP 
for use on POP 11 series corrcuters. 

The general ouroose of OPIuT is to aid decision makers 
bv providing them a capability to construct# store# 
retrieve# exercise# and refine decision-analytic models of 
complex decision problems they face. The (JPINT decision 
model# which is interactive# real-time# and on-line# is an 
organizing framework for information orocessinq. Decision 
analysis is a methodoloaical tool with which the decision 
maker defines and exercises the DPI ^ i T model to evaluate 
decision alternatives oertaininq to the problem (^mey et 
al.# 1979/ OD. 

The OPTMF system is designed to be used interactively by 
decision makers who are relatively unsophisticated with 
resoect to computer technology. Accoroinaly/ the design 
satisfies two human-factors objectives: OPINT is a menu- 
driven system an(d is desianed to be forgivina of oro- 
cedural errors by the user. 
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E^ct^ rlecision mo cl el created bv the user mav be aiven a 
uni cue labels althouqn each is constructed usino the sa>Tie 
Generic format* The model format is shown qrachicallv in 
F i q u r e 16* 

The format consists of the following elements 
which/ when comole tely soecified/ uniauelv define an 0^1 NT 
decision model: 

1. The decision - A short label defining the 
decision problem. This label is also apolied to 
the decision model and is used to store and 
retrieve the model* 

5* Decision Alternatives - A list of decision 
alternatives available to the decision maicer* 

Eacn alternative is aoorooriatelv labeled* 

5* An Uncertain Future Event - A key uncertain 
event/ E/ that will influence the eventual 

outcome of the decision* The uncertain event 
is attached to each of the decision alternatives. 
Event Outcomes - A list of the discrete event 
outcomes/ each aoc r oo r i a t e 1 y labeled/ that 

together define the universe of nossibilities 
reoardina the occurrence of the future event. 

5* Event P r obab i 1 i t i e s - A vector of orobah i 1 i t i es 
that are associated with the event outcomes/ 
which represent the orooability that an event 
will occur* A o’robabilitv is a measure of 



uncertai nty 



It is a number between 0 and 1/ 



DECISION 



DECISION 

CHOICES 



KEY 

UNCERTAIN 

EVENT 



EVENT DECISION 

OUTCOMES OUTCOMES 




FIGURE 16: DECISION MODEL FORMAT 
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inclusive 



that reoresents the extent to which 



an iniiviOual believes a future event will occur. 
Howevf^r^ in this soecificatioPf QroD<3hilities 
are exoresseo as a oercentaae of certainty^ 
e.q./ as U0% vice 0.^4, 

b. Decision Outconnes • The elements discussed thus 
far ciefine oossible decision outcomes. Each 
decision outcome is a oair^d combination of one 
decision alternative with one event outcome. 

The remaining three elements of the model format are 
used to soecify thP conseouences associated with 

the decision outcomes. The consequence of an outcome is 

exoressed in t«^rms of the relative regret that would be 
exoerienced by t^»^ decision maker should the outcome 
actually occur. These three elements are: 

1. Decision C*'utccme Criteria - ^ list of criteria^ 
each aoorooriately labeled/ by whicr the 
decision maker would judge the relative regret 
associated with the decision outcome. 

2. Criteria .n eights - A vector of weights 
associated with the criteria/ which represent 
the relative contribution of the criterion. 
Criteria weights are expressed numerically as a 
oercentaae of the whole/ e.a./ as 60%. 

3. Reoret - Regret is a measure of the conseauence 
of a decision outcome. The total regret 
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assionea to e ciecision outcome is a weighted 
linear corrioin^tion cf the individual criteria 
rearets* For each criterion ana for each 
ciecision outconne^ the user must soecify a value 
of regret. A regret is a nLjmber between 0 and 
•1/ inclusive/ that reoresents tne relative 
decree of dissatisfaction that the decision 
maker associates with a particular decision 
outcom^ii. Zero represents no recret; -1 
represents maximum regret. However/ in this 
specification reorets are always expressed as a 
percentaae of the maximum; e.g./ as -30 vice 
-0.5. t^efer to Figure 17. 



This completes the model format. The decision model is 
completely and uniquely specified when the elements 
described above are defined by the user. The input 
soecifications descrioinq the model can be processed to 
orociuce the following results! 



!• Comoined Value Recret s^atrix - A single matrix 
that displays the total or combined recret 
associated with each of the decision outcomes. 
For each outcome/ the combined regret is 
obtained by weighting and adding the component 
rearets contributed by each criteria. A combined 
value recret matrix is snown in Figure 18. 

2. Expected Value '^’atrix - A single matrix that 
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CRITERION: C. 
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EVENT OUTCOMES 
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’^ln2 
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FIGURE 17 : 



REGRET MATRICES 

(Amey et al., 1979, P.9) 
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COMBINED VALUE 







E^ (LABEL) 


E2 (LABEL) 


E3 (LABEL) 




(LABEL) 


0 


-17 


-31 


h 


(LABEL) 


-9 


-9 


-17 


h 


(LABEL) 


-2A 


-22 


-22 


Da 


(LABEL) 


-30 


-26 


-26 



FIGURE 18: A COMBINED VALUE REGRET MATRIX 

CAmey et al., 1979, P* 10) 
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Hi Sclavs the weiahteci exoecteH reqret/ 

associateH with each Decision outcome. The 
exoecteo value matrix takes into account the 
relative likelihooas of the event outcomes. 

5. Excected Value Vector - A \/ector that aisolays 
the weiohted exoecteH rearet# associated with 
each the ciecision alternatives. 

The exoecteo value matrix anH t^e exoecteo value 
vector (total) are disclayeH toaether, as shown in Figure 
1 ^. 

There are two sensitivity analyses that are useful to 
the user* Both are based on the exoecteo value vector ana 
are described as follows: 



1 . 


Threshold ' l a t r i x 


A 


matrix that disclavs 


the 




eleven 


excected val 


ue 


vectors 


Generated 


by 




e i then 


one of the 


f 0 1 lowing 


ooerat i ons^ 


a t 



the user’s ortion: 

a. Varyinq the orobability ot a oesignated 
event outcome^ from 0 to 100%^ in steos of 
10* The other event outcomes^ maintain 
their proportional relationships with each 
other. 

b. Varying the weight of a designated criterion 
from 0 to 100 /(r in steos of 10. The weights 
of the other criteria maintain their 
orooortional relationship with each other in 
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EXPECTED VALUE 







E^ (LABEL) 


E2 (UBEL) 


E3 (LABEL) 


TOTAL 




(U\BEL) 


0 


0 


-A 


-4 


D2 


(LABEL) 


-5 


0 


-2 


-7 


% 


(LABEL) 


-12 


-1 


-3 


-16 




(LABEL) 


-15 


-1 


-4 


-20 



FIGURE 19: 



AN EXPECTED VALUE MATRIX AND 
AN EXPECTED VALUE VECTOR (Amey 
et al . , 1979 , P . 12) 
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.. it V'.uv » I”.* l/U<St 

(fl .1 ,PTtl i» •% 



the s ^ '^e 



manner as do oroo abilities 



Oescrit)A(i in t^e orecedino oaraaraph. 

In PotH of thp above? caseS/ the least rearet disolaved 
in the excecteo value oayoff vector is identified by 
an asterisk. 'JormaHv# the decision alternative that leads 
to the decision outcome having the least rearet will change 
as the iesianatea event probability or criterion weight 
is incremented from 0 to 100%. The coints of change 
are referred to as threshold ooints and are noted on the ma* 
trix. Fiaure -?0 shows an examole threshold matrix. 



EXPECTED VALUE WHEN 
PROBABILITY OF E^ (LABEL) IS; 

0 10 20 30 AO 50 60 70 80 90 100 
(LABEL) -57 -52 -A6 -AO -3A -29 -23*-17*-ll* -6* 0 

D2 (UBEL) -51 -A7 -A2 -38 -3A -30 -23*-21 -17 -13 -9 

D3 (LABEL) -27 -27 -27*-26*-26*-26*-25 -25 -2A -2A -2A 

Di, (LABEL) -26*-26*-27*-27 -27 -28 -28 -28 -29 -29 -30 



FIGURE 20: A THRESHOLD MATRIX CAmey et al . , 

1979, P. 15) 
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?. '•■'enuallv c'lanae event orohaoilities - The user 
nev nenerate a test exoecteo value vector that 
is bas^o on an arbit-rarilv assioned vector of 
event c robab 1 1 i t i es . The user Tay soecify 
several different orooabilitv vectors and note 
the resultant e’foecteo value vectors. An 

exanole of the disolav is shown in Figure 21. 



KOREAN INTENT 

NO ACTION HARASS SHOOT DOWN 

CURRENT LIKELIHOOD: 60 30 10 

ENTER REVISED VALUES: 30 60 10 

NEW VALUES; 30 60 10 

IF THESE VALUES ARE CORRECT TYPE GO: (GO) 

EXPECTED VALUE/REGRET 



DO NOT FLY -38 
MODIFIED ROUTE -26 
ARMED ESCORT -10 
HIGH PERFORMANCE A/C -10 
NORMAL MISSION -29 



FIGURE 21: MANUALLY CHANGED EVENT PROBABILITIES 

(Amey et al., 1979, P. 16) 



OPINT is o®signed to oerforn the basic functions 
descrioed below: 



1. "Maintain a Horary of QPIMT models - Store 
various decision models, filed by their 
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associated labels 



?. Loar^j an existina DPI'^JT model - Disr^lay t-he 
labels of those monels store ci in the model 

liorarvf and oermit the user to retrieve anv 
nesired model. The loaded model is referred to 
as the current model. 

3 . Pisolav the results of the current mode! - 
Permit the user to examine the structure and 
content of the current model by disolavira: 

a. event o robab i 1 i t i es / 

b. criteria weiahts/ 

c . renre t s f 

d. combined value matrix^ 

e. exoected value matrix^ and 

f. exoected value vector. 

Pevise the current model " Permit the user to 
make chances to the structure and content of the 
current model. The user may revise! 

a. event or obar i 1 i t i es f 

D. criteria and criteria weights^ 

c. decision alternatives^ 

cl . regre t s f and 

e. combined value regret matrix. 

5. Save the current model • Permit the user to add 
the current model to the model library. 

6. Perform sensitivity analyses “ Permit the user 
to test the sensitivity of the current model by 
d0termininQ thresholds or manually changing 
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probab i 1 i t i es 



7. Create new OPI ' i f toopI - Pemit the user to 
create a ne>^ no del/ which then beconnps the 
current nouel. Tne user creates a nodel by 
sceci^yinfa the elements required for the model 
as exolained earlier. The user may also use a 
reference aamhle to validate the assianed values 
of reqret. 
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V, FHF HESIGN of the EXPERIMENT 



The conduct of exoeri^ents using cor^mand and control 
decision aids^ such as OPINT, present special nroblems (Cain 
and Poh^ 1P78, p. p). Historically# the evaluation of 
comrrand and control svste'^s has been focused on hardware 
Teasure='nent3* i. irtlp att<=‘ntion was Paid to software# 
esoec Tally decision a^^ds. a result# few well defined 
Quantitative reasurenents exist for measurino the quality of 
C5 decision aids (Sinaiko# 1^77r oo. 5*o)« Seconn# there 
are no ** correct” answers to a decision problem (Daniels# 
IP77, n. tb), R^fher^ an aid such as OPIfiT is a cognitive 
tool to assist the decision maker in revealina and 
formulizina the elempnts of the decision problem as the 
decision maxer oerc^^^ives them* Lastly# during the 
develocment of these decision aids# there has been 
relatively little synergism between developers and potential 
users (Cain and Poh^ 1978# o. 6)* Users are often not 
familiar with the new technology and are unsure exactly what 
to exoect in the way of performance. Developers are unsure 
exactly what the potential users want. This occurs because 
they aren’t sure who the users will be and causes a very 
general aid to be developed • on^ which will try to suit 
everyone's needs. 

Any experiment v/hich involves the use of decision aids 
must be designed carefully. The experiment should be 
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desianea in nmer to assess both the caoaoilitv of the 
technology and its overall utility as it relates to total 
syste-T oerfornance (i^^anielSf 197 7 / o, There are four 
levels of e^ceri’T’ents varvino in forTalisT fron very loose 
"free olay" with only subjective judgment cutout to "highly 
structured" with complete soecification of conduct of trials 
and out’Dut consisting of carefully measured system 
attributes (^arr et al./ 1^7^/ o* 2 ) * These levels from 
least structured to most structured are: 



1 . 


V a 1 1 d a t i o n 


experiments. These 


experiments would 




consist 


o f 


the 


deouaa i ng 


of ha r dw a re or 




s o f t w a r e . 


The 


t ea s i 


b i 1 i t y of 


the system to work 



accordina to fixed soecifications is determined. 
An example would oe the debuaaing of the OPI^iT 
software orior to placing it on line for 

ooerational testing or use. 

?. Demonstration experiments. These experiments 
would be somewhat more structured than the 
validation experiment/ in that a scenario would 
be followed. Fhe term demonstration refers to 
the type of output from the experiment. The 
output would be mostl’Y personal impressions in 
the minds of the users of the system under 
demonstration. 

3. Assessment experiments. These are experiments 
where triads are conducted over a wide range of 



p7 



|pPK?£ ^^«|||||^^ 







c o n i 1 t ions 



with little control over sources of 



error. Tne qoel is to obtain an idea of ho>^ 

^ell the svste^^ oerforms. The output is 
subjective opinions of the e x oe r i rren t e r s and 
subjects. Their opinions of how worthwhile the 
system is and any potential uses are recorded as 
experimental data* 

'4. Evaluation experiments* These are the most 
riaorous tyoe of experiments. Experimental 
conditions are carefii lly controlled* ^ number 
of replications are oerformea followed by a 
formal analysis of numeric measurement data* 

The assessment experiment was chosen as the means for 
determining tne utility of OPINT. No prior experiments 
concernina the utility of DPI NT have been performed and 
therefore no baseline currently exists* The results of this 
experiment should fill that oao. Subjective impressions and 
judgments of C3 users are what is needec:! to help cietermine 
the usefulness of OPRiT and that is what this assessment 
will do * 

The major objective of this experiment was to assess 
the utility of OPINT as a computerized aec i si on aid in C3 
applications* Secondary objectives which helped in this 
ooal included tne followino: 

1* To assess tne extent to which OPITiT facilitates 
thorouah an.j timely decision makina* 
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2 



To invest! q ate 



the user-O^INT interface and to 



suaaest rer^edial measures or alternate cJesiqns 
as aoorooriate, 

. To survey user ooinion on tne oceration c^nd 
ootential usefulness of QpPiT in C3 decisions. 

The oriTiary objective of OPINT is to assist decision 
mai<'ers in the structurino and analysis of decision problems 
(Amey et al.r \ ^7 ^ f o. ^). One could imply from this that 
a better decision or oroduct is delivered thro u oh the use of 
OPI'IT. The question which is now immediate iSf '*What is 
better?’* Better can not be scenario dependent or olayer 
deoendent. It must be general in nature. DPI NT's 
performance may be be measured in terms of time (the time 
from Perception of oroblem to decision). A faster decision 
is often times needed and therefore the capability to do so 
may he termed '*aooa to have”. But it is only "good to have” 
if the quality of the decision remains at least as good as 
without the aid. In order to assess this feature/ a method 
for measuring the quality of the decision is needed. As 
stated earlier/ the '* correctness" of a decision is extremely 
difficult to measure if at all. In this experiment no 
attempt was made to define or measure this attribute. 

‘A’ hat remained was the subjective judgment of the 
subjects involved in the experiment. The subjects were 
students from th^ C3 curriculum at the Naval Postgraduate 
School. C3 St ij dents were used since the assessment was 
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oerfoTTe^ of GPI^iT*s ahHitv to help with C5 tyoe oroolems. 
These stuoents ^/ere all ooerational officers with a larqe 
amount of pxcerience in ooerational nroblens in the field. 
They reoresenteo the Air Forces Navy^ and *^arines. 
They were in their last (sixth) Quarter of studies and had 
been exooserj to numerous oecision models and aids. The 
subjects were qrouoed into oairs to form a decision team. 
All tne subjects had been trained in the theory and 
ooeration of O^T'iT ana had received at least three hours of 
instruction incluaina one hour of hands-on time. 



Two 


types o 


f decision si 


t ua t ions 


were used - 


time 


critical 


ana time 


not critical. 


It was 


understood that 


the 


ranqe of 


time possibilities is i 


n f i r 1 t e 


and there was no 


way 


to assess 


each. 


Therefore/ the 


two extremes that could 


be 



encountered were useo. 

Two scenarios were followed which reouired the subjects^ 
actinq as decision makers/ to recommenci a specific course 
of action. The tv^o scenarios represented varying comolexi- 
ties which ^ace a C3 decision maker. They were carefully 
designed to avoid the allegation that they were either OPINT 
or subject dependent- The first scenario/ a Cuoan 
blockade problem/ was rather complex. It required an 
in-deoth assessment of intelligence estimates in order to 
predict the occurrence of an uncertain event. In ad- 
dition/ the evaluation of numerous courses of action was 
requireci. The second scenario was an air reconnaissance 
oroblem. This one was somewhat easier as the kev uncer- 
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tain event hen elreedy b^^en assesses and the decision r^aker 



was only reauired to evaluate the courses of action which 
were availaole, cooy of eacn scenario is included in Ad- 
oend 1 X ^ . 

The subjects olaved one scenario usinq OPINT and tnen 
olayevd the second in a manual mode without the use of OPINT. 
Fhe schedule was varied so some subjects plaved the dif- 
ficult Cuban orohlem first and some the easier reconnais- 
sance orcolem. In adoition/ the use of OPINT was varied 
so it was useci alternately first or second. Figure 2 .? ocr- 
trays a tycical schedule. Notice the added di "tension of 
the time critical and time not critical factors. 

It was recognized the evaluation measures used in the 
experiment would not be Quantitatively measurable. 
Therefore care was taken to carefully select both measures 
of effectiveness (MOF.) and measures of oerformance (MQP) 
which suDoorted the goals of the exo^riment. 

A measure of effectiveness is defined as those 
evaluation measures which indicate the contrit)ution of OPINT 
to the overall decision orocess (ACCAT Qoerational 
Evaluation Task/ 1^78/ o. 6). Measures of oerformance are 
defined as those evaluation measures which indicate a level 
of technical oerfornr. ance relative to the internal 
functioninc of OPI^iT itself. Both are aoolicable to 
decision aid experiments. The I'-'OE’s and ''-OP’s were desianed 
so as to: 

1. Pleasure the contrib:)ution of OPII^IT to the 
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TIME NOT CRITICAL 



CUBAN 

SCENARIO 



RECON 

SCENARIO 



CUBAN 

SCENARIO 



RECON 

SCENARIO 



FIGURE 



OPINT MANUAL 



TEAM 1 (FIRST) 
TEAM 4 (SECOND) 


TEAM 2 (FIRST) 
TEAM 3 (SECOND) 


TEAM 2 (SECOND) 
TEAM 3 (FIRST) 


TEAM 1 (SECOND) 
TEAM 4 (FIRST) 



TIME CRITICAL 
OPINT MANUAL 



TEAM 5 (FIRST) 
TEAM 7 (SECOND) 


TEAM 6 (FIRST) 
TEAM 8 (SECOND) 


TEAM 6 (SECOND) 
TEAM 8 (FIRST) 


TEAM 5 (SECOND) 
TEAM 7 (FIRST) 



22: EXPERIMENT SCHEDULE 
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decision process f not the decision outcome. 

2. Be objective so thev do not introduce bias. 

3. Pe simple, to achieve economy and ease of use. 

Permit determination of which functions within 
the decision orocess are aided and under what 
conditions. 



5. Yield results in the experiment which are 

extendible to real world ooerational 

envi ronmentS. 

6. Be operationally olausible and intuitively 
understandable by potential users (ACCAT 
Operational Evaluation Taskf 197P, do. 8*9). 



The following is a list of the ^^0E*s which were chosen 
tor use in the experiment! 

1. Time elansed in arri vina at the decision. This 
includes the time from perception of a problem 
until a decision is reached. In addition/ the 
time elapsed in the actual use of the aid will 
be measured. 



2. Identi fi cat ion/el irination 
information. 



of non-essential 



3. Objectivity in determining relationships among 
factors contributing to the selection of an 
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alternative course of action 



Pisk oerceiveo to oe associated with the 
decision. 



5. Ease of use of the decision aid. 



6. Ease of modification of data within the model. 

7. Perceived advantaoes/disadvantaoes qained from 
the use of th^ decision aio. 



8. Ease of recovery from user errors. 



Data ^as gatherea on each of the abovef throuab the use 
of Questionnaires. The exceotion was the first l^OE. For 
this one f times were r«>corded while the subjects were using 



OP I NT 


t 0 


c aot u r e 


the amount 


of time required 


t 0 


b u i Id 


the 


model f 




time to 


evaluate 


the 


da t a f and 


total 


time 


t 0 


d e c i si 


on 


. These 


times were 


then 


compare (j 


with 


times 


t o 


evaluate 


data and total 


time 


for the manual 


operation. 


Three 


ques t i onna i 


r^s were used to 


reco rd the 


c 

Q 

O 


on s of 


the 



subjects. ^ questionnaire was comoleted after the play of 
each scenario and one conoletecJ after both. Samole 
questionnaires are included as Appendix C. 

Ourino the actual olav of the scenarios^ a 
consideration was how to satis'^actorily place the subjects 
into the mental role required. Tnis included the ability of 
the subjects to adequately assess orobah i 1 i t i es of future 
events anq the renrets required in the construction of the 



model. This research design subsumed this consideration by 
locatincj orodability estimates within the scenarios and 
aefinina the ooals and obiectives to be reached. Subjects 
were encouraoe'i to [-irovide their own reoret assessments if 
they had exoerience in similar ornt)lems. If the subjects 
had no previous experience^ an individual was available to 
act as a senior official. The senior official’s role was to 



o r 0 V i d e 


reoret datar 


i f 


neededf and to 


answer 


cjues t ions 


conce rn i 


no uncertaint 


i es 


or ambiauities. 
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VI. the results of the EXPhRIMENT 



Tnree significant results from the exoeriment were 
revealed. First/ nearly all (85%) subjects stated they 
liked the caoabilities which OPIMT aave them and would use 
0PI?^?T in the future if it was available to them. Second/ 
although they liked the caoabilities of f)PINT/ all (100%) 
stated they die riot like it in its oresent form/ 
particularly in the area of user- interface. Finally/ a 
oroarammino buo (error) was discovered in the use of the 
sensitivity analysis. The sensitivity analysis permitted 
the comoarison of only three courses of action/ showing the 
expected regrets for tre others as zeros. 

It is imoortant to keeo each of these results in mind 
as the scecific results are read. Wany negative comments 
were conditioned oy the statement that if user-interface was 
better/ or if the sensitivity worked correctly/ the comments 
would be Positive. The cases where this occurs are 
appropriately noted. 

The sensitivity analysis orooramminq ”buq” was found 
during the play of the Air Reconnaissance Scenario. This 
scenario had five possible courses of action which needed 
evaluation. OPI'jT handled the expected value and comoineo 
value calculations perfectly. But whenever a sensitivity 
analysis was asked for , 0PI^iT would not compute the expected 
rearet for all courses of action. Aoditional research 
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revealecl that in the case of a sensitivity analysis where 
the oroOaoiHty of t^e event was variedf OPINT would only 
cor»Dute threp courses of action. I'ihen the variation of the 
criterion weiohts was asked for^ OPINT would ccrroute four 
courses of action. Any re^aiTiinq courses of action received 
rearet values of zero. This did not oerrpit a comolete 
evaluation of alternatives and had an adverse effect on the 
fpelinqs of the subjects toward the decision aid. Although 
QPirjT had only c:>^en installed on the POP 11/70 for one month 
prior to the exoerimentf o roo r amm i no buas such as these 
should have been detected and corrected orior to presenting 
the aid as one r a t i ona 1 1 y ready. 

All but four Subjects saivd they felt the scenarios were 
realistic. These four felt that priorities established in 
the scenario were not what they t^lt they shoula be and that 
prior decisions made aid not see^^ realistic from their 
viewpoint. The use of the realistic scenarios coupled with 
the ability of the suoiects to inout their ovm probabilities 
and regrets facilitated a decision making environment that 
was as realistic as possible. 

The eight M0E*s stated in Chapter Five were evaluated 
through the use of questionnaires. A summary of the results 
for each is snown below: 



1 . 


Time elapsed i 


n arriving at 


the 


decision. Times 




were recorded 


in order 


t 0 


capt ure the time 




subjects were 


spending on 


the 


scenarios/ both 



with and without the utilization of OPINT. 
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Those ti^es were recorded not to riet ermine if a 



f a s t e r 


a e c 1 s i 0 n 


was 


Pei no made/ 


hut 


t o 


provide 


an 1 nd 


i c a t i 0 n of 


h 0 1 


ong it took 


t o 


bu i 


Id the 



OPH^^r model and evaluate it. In all caseSf more 
ti'T’e vvas soent while usinq OPIMT than in the 
manual state. The time reouireo to build the 
OPT NT model avera oed ^5 minutes for the Cuban 
scenario and minutes for the Air 
Peconnaissance scenario. The times were tightly 
arouoed as is shown by standard (deviations of 4 
and 2 minutes resoectivelv. The time to 
evaluate THE 0^1 NT model averaged 21 minutes and 
m minutes. Aaain small standard deviations of 
i and 1 minute show tigntly arouoed data. These 
times are very near to what was expected and 
Show a General ability of the subjects to build 
ana manipulate the model. 



Identification/elimination of non-essential 
information. The goal here was to determine if 
the decision orocess was aided by identifying 
non-essential information. 63% of the users of 
OPINT were able to identify some non-essential 
information while only 31% were able to in the 
manual mode. The items of information 
identified varieci according to subject-soecified 
probabilifi^s and rearets. An adoitional ^4% of 
the OPINT users stated problems with the 
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sensit-ivity analysis creventen them from 
oossioly iclent'ifyinq any non-essential 
information. 



3. Oojectivity in Oeterminino relationsHios amonq 
factors contributino to the selection of a 
course of action. The ooal of this MOE was to 
determine if the subjects were aided in the 
decision orocess ny determination of which 
factors were the most critical. 63% felt (JPIiNiT 
definitely aided in the decision orocess. These 
subjects definitely liked the way OPINT oromoted 
detailed and non-sub i active analysis of what was 
aoino to haonen and whv. The rest found it 
confusinq due to uncertainty of what the 
disolays were actually telling them. 6 3% also 
stated they were able to assess the criticality 
of the contributino factors when using OPIfJT. 
The evaluator*s observation here was that in the 
manual mode^ one factor was determined to be 
critical and then the rest ignored. This did 
not occur when OPINT was used. 



Rist< oerceived to be associated with the 
decision. This was used to measure the 
degree of uncertainty or certainty in the final 
decision. 85% of the subjects in the manual 
mode said they made the best decision oossible 






only 56 in the OPIfNjT mode f^lt that 
certain. Two tyoes ot comments were macje. 
First# the sensitivity oroblem was aaain 
mentioned as a hindrance. Secona# several 
subjects felt uneasy about what the numbers in 
the disol ays actually meant. For example# in 
the exoectevd value# combined value# and 
sensitivity disolays# only numbers are shown. 
There is no explanation of what ^hey mean or 
which numders are best. 



5. Ease of use of the decision aid. This was 

used to help determine what tyoe of user-OPIIMT 
interface existed. Only 31% stated that OPINT 
presented disolays in a manner which was easily 
understood. Sensitivity and combined value 
disolays received the most comments as being the 
most difficult to understand. Also# there were 
several comments concerning the inputs reouired. 
f'-’any were not sure what was actually being asked 
for and when to hit the carriage return and when 
not to. Almost all S5% stated the displays were 
generated guickly enough. No problems were 
observed here# even ciuring times of Peak usage. 
Additional comments concerned the use of menus. 
More efficient movement throuah the menu levels 
was wanteci. 
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6. Ease of modification of (fata within the model. 
The noal here to determine how easv it was 

to correc^’ ^he many errors that untrained 
tyoists will make, 6S% stated it was easy to 
maife corrections in the model. The only 
nenative comments concerned the use of the menus 
for this ournose. There existed too much 
retracino throuah menu selections to make 
corrections in the same area or level of the 
model , 



7, Ease of recovery from user errors. This ^OE 
measured how well the GPINT oroaram recovered 
from inout errors. Here tne goal was to 
cetermine if normal tyrina or careless errors 
caused major orohlems, of the subjects who 

made errors felt recovery from these errors was 
DOor, Comments centered around some simple 
mistai^^s which caused major orohlems. During 
initial disolays^ a selection of a choice is 
made by moving the cursor to the desired choice 
and then hittinc the key for execution. 

However in six casesr subjects pressed the *X* 
key followed by a carriage return. This caused 
the oronram to halt and the subjects were 
reauired to beoin again. Several subjects felt 
that nebulous inout cfirections caused several 
errors which coulo not be changed until after 
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the entire ^o<ie] was coToleted. 

B. Perceive Cl advantaoes/disadvantaqes aained from 
the use OPIM. The evclicit consideration of 

all factors through the ororrotion of a aood 
thought orocess was the comment most often 
stated. Several stated they liked being forced 
into a logical thought orocess. Also# the 
subjects felt that OPIK'T orovided them with 
documentation of their decision and woulcJ make 
it easier to oresent tne outcome ana justify the 
conclusion. The d i s ad v an t aae s listed for using 
GPINT as it currently exists centered around the 
u s e r - i n t e r f ac e oroblem. There existed a general 
lack of comfort due to the limited aescriotions 
and assistance from the aid, 

9, An additional Question concerning training was 
asked of each subject. They were asked to 

comment on the amount of training they felt was 
required for DPI Ml usage. The overwhelming 
resDonse was about two hours of structured 
instruction on the theory and use of the aid# 
folio weo by one to two hours of hands*on 
demonstrations. This amount of instruction was 
exactly what each subject received crior to the 
exoeri ment • 
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V L i . C^•jCL^l5I0^S A^|D PF^.^MML^'0 A T I QNS 



It was oeternnineci that OPIWf is beneficial to the 
decision ma‘<er in thp decision makina orocess. The majority 
of the subjects comments were either oositive in relation to 
OI^InT or ^^ould hav^^ been with either no program errors or 
better user-interface. OF^Ii^iT’s greatest advantaae lies in 
two areas. Firsts it ororrotes (and oossibly even forces) 
the user into a souna thotjaht orocess. In the heat of 
crisis or even just day-to-day decisions/ it is often too 
easy to foroet the orincioles of good decision making and 
slic to sometnina less. The use of OPIf^iT could help to keeo 
tnat from haooenina. Second/ OPI'^'lT orovides its own 
documentation. This documentation is then available to 
ore sent decisions and show why a oarticular decision was 
made. It is also easy to focus on the critical areas and 
not waste time on the non- important ones. 

There are still some major flaws which need to be 
worked out of the system. The first and foremost is the 
sensitivity analysis. Program bugs such as these can not 
e^ist in an aid if it is to be used orooerly. Second/ is 
the user - interface problem. The aid must be designeci for 
the casual user/ one who at one time may have been 
oroficient/ out throuoh lack of use has forgotten the key 
requirements. ^''iiditional comments concerning wnat is needed 
or what is being Hisolayed are a must. 
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Th<^ 'Movement ljspt f'nrouoh the aia via the 



menus 



also neeos attention. U sens shoulo be niven the choice of 
returnino to several levels rather than only ud one or down 
one. 

Tne final and by no means the least imoortant is the 
oroblenn of orooram halts. ^Mcthing arieves a user more than 
havina to start ov^r# especially after a great deal of work 
has already oeen done. The system currently displays 
numeric incuts for verification. There is no reason why 
alphabetic or menu choices could not also be done. 

This author has Several recommendations for both 
imcrovina the (decision aid and for future experiments with 
it. As stated earlier/ the aid needs to be designed for the 
optional user. This user's knowledce of the aid could range 
from complete to nothing deoendinq on amiount of usage. The 
aid should oe designed ^or tne two extremes and even for the 
user in tne midcile. It needs to be both user friendly" and 
"error tolerant". One techniaue could be optional 
verbosity. For each 'iisolav/ three sets of comments could 
be prepared. The decision maker then selects which set is 
desired deoendinq on need. The knowledgeable user would 
receive only short phrases which could be keyed on to 
generate incuts or disolav results. The same tvoe of 
canabilitv wouM exist for the other tyoes of users. This 
way the user recf=^ives only what is neeaed. Another 
variation of this is to allow each user to specify exactly 
what co'^ments are needed to insure correct usage and 
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interor^^tat ion 
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t^at nee'j. User confidence could imorove areatly with this 
simole adoition. 

Once 0^1 Ur is redes iqned to neet the needs of the 
users/ as stated in tho results/ additional exoeriments 
should oe conducted ^o Teas u re additional capabilities of 
the airi. This evoerinent did not consiaer whether a better 
decision was beinr] made. This is not an easy problem and 
further researc*^ and experimentation is called for. 

Another area for exoeri mentation has to do with the 
area of rearet values. OPF^'iT currently is desianed to 
handle rearets rather than cains. Some subjects casually 
stated thev miaht have oreferred to use gains rather than 
rearets. Additional experiments to determine if the use of 
Gains vice regrets has an impact on user efficiency is 
definitely callen for. 

Finally/ thouoht Should he niven to providing a hard 
cooy notion to users who have only a soft copy (CRT) 
capability. Further e x oe r i men t a t i on could be done to 
determine if potential users want or even need a hard cooy 
c acab i 1 1 t y . 
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bvtes ano ^nini'r>al attention on the user. The use of 
decision suocort systems suc*^ as f)PI'jTr off^r an oonortunitv 
that IS both trivial ana iT^ense. It is iTrrense in that 
they can eTioeo th<o computer in decision .rakina activities 
where larqe oayoffs coujld be provided and trivial in that 
they reores^^nt no major advance in technoloqy. Buildinq 
innovative systems will be a difficult and risky venture for 
some time to come. Toe future of decision suooort systems 
and ooerational decision aids deoends on some adjustments 
that both the develocers ana users must make. But these 
adjustments are small ones# and are mainly attitudinal. 
Little new knowledae is required. But the oooortunities are 
all substantial/ ano the Personal and organizational rewards 
h 1 G h . 
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APPEMOIX A 



SAMPLE OPINf vSESSKV'i 



% o o 1 n t 

1 ) Ann A rbo r 
P) Anv hardcoov 
3) AOM 3 A 

'^hich terminal tyoe are you usina? P 

Pleese select one of the followinq ootions, 
OPTION 

1 Oisolav results 

2 Edit- Tiodel 

3 Sensitivity 
^ Load model 

5 Create or ajd to model 

6 Save model 

7 New information 

8 End session 
Enter desired ootion:5 



Is this a new model?(v or n):v 

Please enter a title for this model: [air reconna i ssance 

'.''ihich Dortion of the model would you like to create? 
OPTION 

1 Generate influence iiaqram 

2 Generate value/rearet model 

3 Add Bavsian indicators 

^ Return to main menu 

Enter desired ootion:! 



v^ihat is the main event of interest? [korean intent } 
korean intent 




) 



Please enter the outcome laoels for the event you have 
called korean intent 



Enter outcome 
Enter outcome 
Enter outcorne 
Enter outc ome 



label : 
label : 
label : 
label : 



[no action ) 
[harassment 3 
[shoot sown] 
[ 1 



Please enter the oroh^oilty of each outco'^e of the 
event vou have labeled korean intent 



no action h a r a s s ne n t s h c o t down 



-60 

N c r m a 1 i z e d values: 

Tf these values arp correct 


30 lu 

o 0 , 0 

t y oe QO : 


50.0 

(qol 


10.0 


/'/hich Dortion of the model 
OPTION 


would you 


1 i <e 


to create? 



1 Generate influence diaqran 

2 Generate value/reoret model 

5 Add Bavsian indicators 

^ Return to main menu 

Enter desired ootion:2 

fou will be asked to orovide the criterion to be used 
in your value judnementf the actions vou may takof ann 
the associated value/reoret matrices for each criterion. 
Please nress return to continue. 

Enter the names for each of your criterion: 



Criterion: [aircraft-aircrew 1 
Criterion: (loss of information ] 
Criterion: [natio^a] influence 1 
Criterion: [recon orogram ] 
C r i t e r i o n : [ ] 



f^iow enter the labels for each action. 



Action: [do not fly ] 
Action: [modified route 1 
Action: [armed escort 1 
Action: [hiah oerf a/c 1 
Action: [normal mission ) 
Ac t i on : ( } 



You will now be asked to orovide the values for 
each of these value matrices. 

Please oress return to continue. 



do 


not 


f 1 


do 


not 


f 1 


If 


these 



PI 



y 



ease enter reqret/value matrix 
ai rcraft-ai rcrew 

no action h a r a s s me n t s h od t 



values are 



0,0 

c o r r ec t 



t voe 



0.0 

go : 



[qo] 



for 

down 

0.0 
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nodi f i e d 


route 


0 


1 

i\j 

0 

1 

o 

0 

1 
1 
1 
1 
1 
1 
1 
1 




nodi f i e d 


route 




0.0 -^ 0.0 


-60 . 0 


If these 


values 


are 


correct tyoe ao l 


(go 1 


armed esc 


0 r t 


0 


-15 -55 




armed escort 




0.0 - 15.0 


- 35.0 


If these 


values 


are 


cor rec t tyoe qo : 


[qo 1 


h i q h o e r f 


e /c 


0 


-5 -15 




h i a h pe r f 


e/c 




0.0 - 5.0 


- 15.0 


If these 


values 


are 


correct t yoe qo : 


lao] 


no rna 1 nn i 


ss i on 


0 


-25 -100 




n o r n a 1 n i 


s s i o n 




0.0 - 25.0 


- 100,0 


If these 


V a 1 u e s 


are 


CO r r ec t tyoe qo t 


f qo 1 




Pleas'^ enter r eo e t / v a 1 u e matrix for 
current information 

no action harassmentshoot aown 


do not f 1 


y 


- 


10 0 -100 -100 




do not f 1 


y 




- 100.0 - 100.0 


- 100.0 


If these 


V a 1 ue s 


are 


correct tyoe go: 


(qol 


nodi f i e d 


route 


- 


50 -50 -30 




nodi f i e d 


route 




- 30.0 - 50.0 


- 3 u . 0 


If these 


V a 1 u e s 


are 


correct tyoe go: 


f GO ) 


armed escort 


<- 


5 -5 -5 


— 


armed escort 




- 5.0 - 5.0 


- 5.0 


If these 


V a 1 ue s 


are 


CO r rec t type qo : 


[gol 


h i ah oe r f 


a/c 




0 

1 

on 

0 

1 

Ul 

0 

1 
1 
1 
1 
1 
1 


— 


h i a h o e r f 


a /c 




- 50.0 - 50.0 


- 50. 0 


If these 


V a 1 u e s 


are 


correct type ao: 


loo) 


n o r m a 1 n i 


s s i on 


0 


0 -100 




no rna 1 nn i 


s s i o n 




0.0 0.0 


- 100.0 


If these 


values 


are 


cor rec t tyoe go : 


loo] 




Please enter reqret/value matrix for 
national influence 

no action harassmentshoot down 


do not fly 


- 


50 -1 00 -70 


— 


do not f 1 


y 




- 50.0 - 100.0 


- 70.0 


If these 


V a 1 ue s 


are 


correct type ao : 


(gol 


modi f i ed 


route 


* 


20 -20 -50 





mod i f i ed 


route 




- 20.0 - 20.0 


- 50.0 


If these 


V a 1 ue s 


are 


CO r rec t t voe ao : 


(ooJ 



HO 



an'T^eri escort 
arnned escort 



-iO 0 0 

-ao . 0 



0 . 0 



these values are correct tyoe co: laoJ 



hiah n^rf a/c 
high oerf a/c 



-?5 -10 -10 

-25.0 -10.0 



0 . 0 



10.0 



If these values are correct tyoe ao: [aol 



normal mission 
normal mission 



0 -25 -100 

0.0 -25.0 



- 100.0 



If these values are correct tvoe qo: [qol 



Please enter reqret/value matrix for 
r e c onn a i s s an c e oroo 

no action harassmentshoot Oown 



do not fly 
do not fly 



- 1 00 -100 -100 

- 100,0 - 100. 0 - 100.0 



If these values are correct tyoe oo : [qol 



modifier route 
modified route 



- 70 -50 -50 

-70.0 -50.0 



-50.0 



If these values are correct tvpe ao: [qol 



armed escort 
armed escort 



0 0 0 -- 

0.0 



0.0 



0.0 



If these values are correct tyoe no: [gol 



hi oh oerf a/c 
h i qh pe r f a/c 



-10 -10 -10 

- 10.0 - 10.0 



- 10.0 



If these values are correct tyoe go: fao) 



normal mission 
normal mission 



-1 0 -70 -80 

-10.0 -70.0 



-80.0 



If these values are corr^^ct tyoe ao: [oo) 

Please enter a set of importance weights for these criterion 

ai rcraft-ai rcrew 

current information 

national influence 

reconnaissance orog 

wts: 100 10 50 30 

Norm: 58.8 5.d 17.6 17.6 

If these values are correct type no: [go) 

v/vhich Dortion of the model would you like to create? 

OPT lOiM 

1 Generate influence diaaram 

2 Generate value/rearet model 

3 Add Bavsian indicators 

0 Return to main menu 

Enter desired option:4 



oo 



Please select one of the foHowina ootions. 

OPT ion 

1 Disnlev results 

2 E 1 1 t Ti o 1 e 1 

3 Sensitivity 
L o e d TH o d e 1 

5 Create or ado to model 

6 Save model 

7 New information 

8 Eno session 
Enter desired ootionrl 

The followinq disolays are available 
OPTION 

1 E vnec t ed value 

2 Comb i ned value 

3 Event likelinood 

''4 Values 

5 Value weinhts 

6 Influence diagram 

7 Return to main menu 
Enter desired option:! 

Expected va1ue/rearet 



donotfly -3S.4 
modified route -23.^ 
armed escort 

hiahcerfa/c -9.8 
normal mission -20.1 



Please cress return to continue. 

The follo-vinq disolavs are available 
OPTION 

1 E xpec ted value 

2 Como i ned value 

3 Event 1 i w e 1 i hood 

^ Values 

5 Value weights 

t Influence diagram 

7 Return to main menu 

Enter desired ootion:2 

Combined Value 



no 


action 


harassmentshoot 


down 


do not fly 


-32. y 


-ai .2 


-35.9 


modified route 


-17.6 


-25.9 


-5^1 .7 


armed escort 


-7.iJ 


-9. 1 


-20.0 


high oerf a/c 


-^. 1 


-9. a 


-15.3 


normal mission 


-1 .8 


-31.5 


-96.5 


Please cress return 


to continue. 
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aval I ao 1 e 



The follo/jina isolava are 
OPTIOM 

1 E X oec t eo value 

2 Comoinp^ valu® 

3 Event liicelihood 
Values 

5 Value w e i a h t s 

0 Inflijence diaoram 

7 Return to main menu 

Enter desired ootion:7 

Please select one o^ the following ooticns. 
OPTION 

1 Disolav results 

2 Edit moael 

3 Sensitivity 

^ Load model 

S Create or add to model 

0 Save monel 

7 New information 

S End session 

Enter desired ootionrS 

Select one of the following: 

OPT ION 

1 Event orooability thresholds 

2 Value weiohr thresholds 

3 Return to main menu 
Enter desired notion:! 

/ihich event outcome likelihood should oe varied? 
OPTION 

1 no action 

2 harassment 

3 shoot down 

Enter desired notion:! 







E xoec ted Value 


when 














Probability of no 


action 


i s : 












0 


1 0 


20 30 


OO 50 


6 0 


70 


80 


90 100 


do not fl 


y 


-39 




-37 -37 


-3o -35 


-30 


-30 


-33 


-32 -31 


modi f i e d 


route 


-32 


-31 


-29 -27 


-26 -20 


-23 


-21 


-20 


-18 -17 


armed escort 


-1 1 


-1 1 


-1 0*-l 0* 


-Q* -9* 


-8* 


-8* 


-7* 


-7 -6 


high oe r f 


a /c 


-10» 


-10* 


- 1 0 * - 1 0 * 


-10 - 1 0 


-10 


-10 


-o 


-9 -9 


normal mi 


s s i on 


-a2 


-38 


-30 -50 


-2b -22 


-IB 


-10 


-10 


-b * -2 



Please oress return to continue. 



S e 1 e c t 
OPTION 
1 
2 



one of the following: 

Event orobability thresholds 
Value weight thresholds 



R2 



5 Return to main menu 

Enter desired ootion:3 

Please select one of the followina ootions 
OPTION 

1 0 1 so 1 ay results 

2 Edit Tonel 

3 Sensitivity 

U Load TOO el 

5 Create or add to ^odel 

6 Save Todel 

7 New information 

6 End session 

Enter desired ootion;6 



Please 

OPTION 

1 

2 

5 

a 

5 

6 

7 

8 

Enter 



select one of tne 

Disolav results 
Edit model 
Sensitivity 
Load model 
Create or add to 
Save model 
fJew information 
End session 
desired option*^ 



f o 1 lowing 



mode 1 



ootions 
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A p p E 0 [ X E 



SA-v!PLE sce^japios 



CUBAN BLOCKADE SCENARIO 

On 25 ■’^'arch 1960/ ohotoaraohs received from military 
surveillance aircraft showed the e>^istence of new 
construction alonq the southern coast of Cuoa. Intelligence 
analysts l:)elieve th^ construction to be a new major Soviet 
nuclear submarine suooort case. However/ they are undecided 
as to whether it will be caoable of resunolying only nuclear 
fuel or botn nuclear fuel and nuclear SLH*^- missiles. 

Additional intel licence from other sources has revealed 
the cresence of hundreds of new technicians in the area. In 
resconse to American inquiries/ the Soviets have stated that 
they are merely neloina the Cubans build a new merchant 
oort. They claim the technicians will leave as soon as the 
oo r t is como 1 e t ed . 

i'^^ost US ooservers are hiahly concerned over these new 
de V e 1 oom e n t s . They feel the oresense of a new oort/ caoable 
of nuclear retrofit for Soviet submarines/ has a serious 
imoact on US national security. No longer would the Soviet 
submarines be reouired to return to home ports. They could 
conceivably oatrol near our shores for unlimited periods of 
time. If a nuclear war ever did break out/ a missile 
resuDoly base so near—bv could be devastating. 

Analysts oelieve the new construction is caused by the 
oossibilitv of non~ratification of the new SALT Treaty. 
They feel if the Treaty is not ratified/ the Soviets will 



9a 



have a jumo in arT. s escalation and the US will have no cower 
to interfere. The analysts also Del i eve that if the Treaty 
is ratifiedf tne construction will Quickly cease. 

On the ^ornina of 21 ^'»arch, three Soviet shios were 
oetected sailina towards Cuba, Reconnaissance has revealed 
one to be a fre ianter# one an anred escort destrover^ and 
the last an escort submarine. Intelligence sources in the 
Soviet Union have sai'-l the freighter is carrying the first 
suDcly of nuclear fuel and may oossiblv be also carrying 
SLB^ • s . 

Fears that the Soviets may oe trying to quickly 
establish a stronghold in Cuba Dromnteo the President to 
order a three shio Task Force to intercept the Soviet 
freighter, fhe mission of the Task Force is to determ, ine if 
nuclear suoolies of any type are on board# ana if so# to 
detain it awaiting further instructions. 

The Soviets immediately resconded by statina they would 
absolutely deny any boaroinq of the freighter and any such 
action would be met by military retaliation. Intelligence 
has also revealed the ordering of at least ten Soviet 
submarines into the area and the rapid augmentation of its 
Atlantic Force, 

The President has decided he can not allow nuclear 
supplies to reach Cuba and he feels a bloci<ade of some tyoe 
must be olaced around Cuba. He immediately placed the 
Atlantic Fleet on alert and ordered them to begin steaming 
towards Cuba. In addition# he also placed the Med i t e r r anean 
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Fleet on alert 



T n e t y c e 


and 


5 w e 0 ^ 


blockade has 


not 


vet 


been 


ieterminecK 


The 


Presinenr 


has asked 


you 


for 


your 



reccniTenaation. Ihe President soecificallv wants a 

rec onnTenda t i on as to whether a total olockadp all 

shiDDinq/ a nriodified blockane some sort allowing merchant 
shios carryina non-military caroo to oasS/ or a roving 
oatrcl which woula only detain ships carrying military 
h a r dw a r e • 

Ooerations oersonnel have stated it would require the 
entire ^^tlantic Fleet ana half of the ‘^Mediterranean Fleet to 
invoke a total blockade, A modified blockade would require 
only the Atlantic Fleet while the roving oatrol would only 
need half of the Atlantic Fleet resources, 

Intellioence an=^lysts are unsijre what the Soviet 
intentions will be concernina the situation. They believe 
their ootions to be one of three: 

1, An attack of the hlockaainq forces. 

2, An attemot to run through the blockade without 
initiatina any aaoression# or 

5, Submission to the blockade. This means the 
Soviets may relent to having som^ shios searched 
but mav have others return directly home. 

They agrees howeverr that two upcoming events will 
helo to cietermine their motives. The imoendina vote on 



96 



fne salt Treatv and final determination of exactly 
wnat tyoe of construction is heinq aone on the coast ere the 
l<rev issues in essessino their intentions. In aciaitioo/ 
they feel the SALT Treaty vote >cill he effected hy Doth US 
oublic oDinion concerning the Treatv and current world 
ooinion concerninq the situation. The UM Security council 
is scheduled to meet in two days to discuss the situation. 

The analysts have assessed the probability of US ouplic 
ooinion Deino pro-SALT as 70% and anti-SALT as being 30%. 
/'lOrld ooinion is assessed as having a ^0% chance of being 
oro"U‘S# 30% oro-Soviet/ and 30% of oeinq neutral. The 
analysts also currentiv aaree to a 60% likelihood of the 
construction being to suooort full nuclear resuoply/ 30% for 
only ^iaval resuonly (non-armaments)/ and 10% as a new 
c omme r c i a 1 do r t . 

The items tr>e President wants considered in the 
selection of a best course of action are (in order of 
imoortance) : 



1) National security. The oresident feels he can 

not allow the Soviets to establish a suooly base 
in Cuba . 

2 ) Safety of US Nationals currently in Cuba. The 

b3 has recently beaun exportina numerous 

commercial products to Cuba in an effort to 
Doost a sagging economy. At present/ it is 
estimated there are at least 2000 Americans in 
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C ut? a 



3) I'S nub lie ooinion. This is an election year and 
analysts feel there will be oublic outcry if we 
force the Soviets into a military confrontation/ 
but the cuolic will be impressed by strong 
Treasures which force the Soviets into accepting 
our demands. 

•^ ) Cuban reaction. The US has only recently begun 
trade with Cuba end is anxious to continue this 
jf at all possible. Any tyoe of blockade 
imposed around Cuba will have a serious impact 
on these trade agreements. 

5) Acrid opinion. Any confrontation between the US 
and U3SP is undesirable. If the US is 
successful with the blockade/ world ooinion will 
be high. However/ if we orovoke aaaression by 
our actions/ we will lose additional world 
support which is desperately being sought by the 
Presivdent • 
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RECONf'JA ISSANCE SCENARIO 



On S 10 6 Of an unarTie^J ^'ohawk reconnaissance 

aircraft aborteo its Tiission over Korea. It returned to its 
base where t^e crew r^oorted an interceotion oy a 
fiqhter-tyoe aircraft from North Korea. The crew of the 
unarmed --^ohaw^ had not stayed around long enouah to 
determine whether the intercepter was there to harass^ 
shootf identifyf or had just “haroened along” while on a 
routine trainino fliaht. Another reconnaissance flight is 
scheduled for tomorrow ana the commander needs a 
recommendation concerning future fliahts. 

After further discussion/ the commander desires the 
following ootions evaluated in light of what has haooened 
and what the intelligence staff conclude are North Korea’s 
intent ions: 

1. Do not fly into the area anymore. 

2. Fly into the area on a modified route. This 
modifieo route would be less sensitive but also 
less oroductive in collection potential. 

5, Fly into the same area but now with an armed 

escort . 

a . Fly into the area with hioh-oerformance 
aircraft. The aircraft would be less vulnerable 
but would not collect data as well as the normal 
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a i r c r a f t 



5 



Fly a nor^ial 



mission; do not cFanae anything 



from nrevious fliqhts. 

Tho comrpander is mostly concerned with the oossiole 
loss of the aircraft and crew durino the mission. However^ 
ne is also concerned with the loss of prestige or political 
influence if fliohts were stoooed or alterecJ. The loss of 
information ana the cossihle imoact n the r ec onn a i s s a nc e 
orogram in otn^r areas are also important factors. 

The intelliaence section believes it has assessed the 
Dossible intentions of the North ^orean Government on future 
flights. Based on oast situations ana current capabilities/ 



they 


feel 


there is onl 


V a slim 


chance 


(10%) the 


Ko reans 


will 


t ry 


t o 


shoot future 


a i r c r a f t 


down/ a 


moderate 


chance 


(30%) 


they 


will 


harass / but 


D r obab 1 y 


will do 


n o t h i no 


(60%) . 
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SAMPLE QUEST lONNA If?ES 



QUE S T I Q >lf) A I pp FUP TEAM mf.vpepS ;';IThO[JT OPIUT 
N A‘-E : 

R A I'J K : 



TEAM a : 



SCFUAR TO : 

/inAT s'jAS YfjUR RECQMME'vIDE D ACTION? 



i-jMY DIO YOU RECUMMeuP THAT ACTION? 

PLEASE AfviS.'.'ER THE FOlLO-vTUG QUESTIONS AS INDICATED. 

1 . Do vou feel this scenario was realistic? If no^ nlease 
comment in the soace orovioed. 

8 • Yes 

b. No 

2 . /iere you aole to i 1 entity any non-essential information 
d urine: your evaluation of the courses of action? If yeSf 
wtiat was ttiat information? 

a . Yes 

b. No 



I'Jere you able ^o assess the criticality of the factors 
which contributeo to the choice of the recomrnenaed course 
of action? If yeSf which factors were the most critical? 

a • Yes 

b . Tio 



Do you feel confident that your recommendation was the 
"best"/ based on the information nrovided? If no/ whv not? 

a . Yes 

b . I' j o 
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QUEST lUfiN/^ FO^ TEA-' '-ifMefPS i'j[TH OPINT 



i\j A M E : 



RANK : 



TEAv s : 

SCENARIO: 

;AiHAT '-JAS YQIjR RECi}r-*?^'EMOED ACTION? 

I'jHY DIO YOU PECnvMEND TmAT ACTION? 

PLEASE AOS.'.Ek the FOLLO/iING QUESTIONS AS INDICATED. 

1. Do vou Teel tSis scenario was realistic? If no , olease 
consent in tne snace oroviUed. 

a . Yes 

0 , N o 

P. Uere you aole to identify any non-essential infor'nation 
during your evaluation of the courses of action? Tf yes» 
what was tnat infor-nation? 

a . Y A s 

h. 'Mo 



3. Here yoti able to assess the criticality of the factors 
which contributed to the choice of the reconnenried course 
of action? If ves» which factors were the rrost critical? 

a . Yes 

b . N o 



R. Do vou feel confident that your recorr. men Nation was the 
"best”» based on the information provided? If no» why not? 

a . Yes 

D . ’'lO 



10? 



5. Did OPINT present the evaluated data in a Tianner which 
was easily unjerstood^ If no» which oisplavs were difficult 
to understand? 



a . t es 

D . “'*0 



b . Did OPIi\'T (jisclay its results ouickly enouqh? If 
which disolavs were slow? 

a . Yes 

0. Jo 



7 . '.'ias it easy to .tiake corrections in the model? If 
comment on your tjronlems. 

a . Yes 

b . M 0 

8. Did OPT 'IT recover well from any errors you minht 
made? If not» what was the result? 

a » Yes 

b . f'i c 

C. “ I did not make any errors. 



not# 



not r 



have 
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QUESTT1■^JN^^ l^E FU^ AFTER CO^’RLETTNG BOTH SCENARIOS 



NAME : 



RANK: 



TEA M ^ : 

PLEASE ANS/vER THe; FOLLO/vINQ QUESTIOiNS AS INDICATED. 

1. To what extent dir) OPINT aid in your assessment of 
what event was aoinq to haonen? 

a . Con fused 

b . ’Mo h e 1 o 

c. Pei'^Iorceo own if'ieas 
d . C 1 a r i f i e vO 

e . En 1 i nh t ened 

f. Had no need to assess an event 

P. To what extent did OPINT aid in your decision process 
for select ina a course of action? 

a . Ignored 

n. Relied upon it 

c. Added confidence 

d. :'Jas confusinq 



3. Do you think tnat 0PI^)T helped to speed un the decision 
making crocess? 

a . Yes 

b . No 

^4. Do you think you would have performed better with or 
without 0P^^lT? 

a . . i t h 



P 



i t H 0 u t 



5. Do vou feel that OPImT cou) ^ be imoroved uoon? If yeSf 
in w h i c n a r a s ? 

a . V e s 

h . '«] o 

6. Do you feel decision rpa^^ers shoulo use CPINTf or shoula 
it be left to technical e^oerts? 

a • Decision ^ak<=^rs 

b. Technical exoerts 

7. '/^hat no vou ce^ceive as oeino tne areatest an vantage in 
u s i n a DPI ' ! T ? 



6 , lAihat do you nerceive as h^ino the Greatest disadvantaae 
in using 0 ^ 1 ‘''JT? 



P • ‘/i 0 u 1 d you 
a V a i 1 at) 1 e to 



use 0 ^ I ^'1 T in 
vou? Please 



the future if 
c onnTien t . 



\^as made 




r- 




oi • ( O'! I ' 






♦ 



,# ‘ 1 - 4 






I ^ 1:. e f^h ^ 






»# f ftn: 



• m.i 



f 



^ *ti>**ir 



• f » i • « f i| Y * # t 

ft I f • • 4 %, 
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